Configuration options and parameters

There are many options and parameters you can use when running the storpool_cg tool with the conf command. When the command is run successfully, the resulting configuration is saved in a configuration file.

Configuration options

When running the storpool_cg conf command you can use the options listed below.

--clean: Reset all saved options
-C / --configfile: Configuration file for the tool.
-E / --expand: Allow the migration to increase the cpus count for StorPool. Off by default.
-i / --interactive: Migration interactive mode. Use together with -M. Off by default.
-L / --preserve-limits: Preserve existing memory limits when migrating. Off by default.
-M / --migrate: Apply the configuration live. Off by default.
-N / --noop: No operation, only print the configuration; see Understanding results for details. Off by default.
-P / --preserve-cpus: Preserve cpus and cpu bindings when migrating. Off by default.
-v / --verbose: Verbosity level. Default is 0.

When upgrading a machine (not creating the control groups configuration for the first time) you may utilize the following options:

-P (--preserve-cpus) will read the cpuset configuration for the storpool.slice (only) and will keep it unchanged. Some cpuset subslices of storpool.slice might be deleted during the migration (because they are not needed in the new configuration), so their cpuset won’t be preserved.
-L (--preserve-limits) will read the memory limits of the storpool (and its subslices), system, user, and machine slices and will keep them unchanged. This option will also try to preserve the memory left for the kernel.

Saving a configuration as a file

Tip

This functionality is available starting with StorPool version 21.0 revision 21.0.576.6d56902e2.

When the storpool_cg tool is run successfully, the resulting configuration is saved in to the /etc/storpool/cgtool.ini file.

Warning

Do not edit this file manually! The configuration should only be changed by running the storpool_cg tool.

As described in Loading configuration from a file, it is possible to apply a set the parameters stored in a file by invoking storpool_cg with the \-\–configfile option. The configuration would be loaded from the specified file, and would be saved as usual to cgtool.ini.

Note that setting an option to its default value is different from resetting it. To illustrate this: numa_overflow=0 (which happens to be the default) is different from numa_overflow=. The first will explicitly set the option to 0 and save that in the “.ini” file, while the second will not.

Resetting all saved parameters

The “–clean” option will reset all parameters except the ones explicitly given on the command line. It is equivalent to invoking storpool_cg with “parameter=” for all parameters. Here is an example:

# storpool_cg conf --clean SYSTEM_LIMIT=6G
Resetting CONVERGED
Resetting BLOCK
Resetting ISCSI
Resetting NVMET
Resetting MGMT
...
Resetting SP_CPUS_NOT_EXCLUSIVE
SYSTEM_LIMIT=6G saved
CONVERGED reset saved
BLOCK reset saved
ISCSI reset saved
NVMET reset saved
MGMT reset saved
...

The -–clean option can be used with –-noop to check what will happen. In such a case the last messages will be slightly different, as shown in the example below:

# storpool_cg conf --clean --noop system_limit=6G
Resetting CONVERGED
Resetting BLOCK
...
SYSTEM_LIMIT=6G would have been saved
CONVERGED reset would have been saved
BLOCK reset would have been saved
...

The –-noop option will never save or reset options.

Viewing the configuration

You can review the current cgroups setup by reading the ``cgtool.ini`` configuration file directly:

# cat /etc/storpool/cgtool.ini
[cgtool]
USER_LIMIT=4G
SYSTEM_LIMIT=8G
CORES=8
...

You can also do this using storpool_cg with the \-\-noop option.

# storpool_cg conf -N
Loaded USER_LIMIT=4G
Loaded SYSTEM_LIMIT=8G
Loaded CORES=8
...

Loading configuration from a file

Having a specific set of configuration parameters stored in a file can be helpful when you need to tweak them, and also in cases like the one described in Configuring multiple similar machines.

Assume you have a file with a name like my-cgconf.cfg. You can simply run storpool_cg by telling it about the file:

storpool_cg conf --noop --configfile my-cgconf.cfg

You may override (or specify) parameters in the file from the command line by listing parameter=value pairs. Here is an example:

storpool_cg conf --noop --configfile my.cfg cores=4 converged=1 iface=sp0

This command would set 4 cores and the sp0 interface to be used, and will make a converged setup no matter what are the values (if any at all) of the CORES, CONVERGED, and IFACE parameters in the configuration file (explained below).

Configuration parameters

As shown in the example above, the configuration file for the tool consists of a single section named cgtool. It can contain the parameters listed below. Note the following:

As described in Format, these parameters can be set on the command line as parameter=value pairs.
Parameters’ names are case-insensitive.

BLOCK, ISCSI, MGMT, BRIDGE: Boolean. Specify whether the corresponding Storpool service will be used on the machine. If not specified, will detect which services are installed and set them to True. Consider that the services should be enabled after the configuration of cgroups is completed; for details, see Managing services with storpool_ctl.
CACHEDIR: Path to the directory where server cache configuration file (cache-size.conf) for StorPool servers will be written; default is /etc/storpool.conf.d.
CONFDIR: Path to the directory where control group configuration files will be written; default is /etc/cgoconfig.d.
CONVERGED: Boolean, default is False. Specify whether the machine is hyperconverged or not; cannot be detected.
CORES: The number of cores dedicated to StorPool services. Note that if the machine supports hyperthreading then 4 cores = 8 CPUs, and if not then 4 cores = 4 CPUs. For hyperthreaded machines you may specify for example 3.5 cores, which are 3 cores (6 CPUs) plus one thread (CPU) from a different core.
IFACE: Network interface for StorPool. Should be a machine raw device, for example enp1s0f0. If not specified the value from the storpool.conf will be taken.
IFACE_ACC: Boolean. Specify whether the network interface should use hardware acceleration. If not specified it will be detected. Use this with care, only for overwriting the detection.
KERNEL_MEM: Non-negative integer, use one of the M or G suffixes; default 1G. Specify the amount of memory that will be left to the kernel. Note that KERNEL_MEM will have effect only when some of the other limits have to be calculated (are not explicitly specified).
MACHINE_LIMIT: Non-negative integer, use one of the M or G suffixes. Specify the memory limit for the machine slice.
NUMA_OVERFLOW: Boolean, default is False. Specify whether the usage of non-interface-local CPUs for StorPool is allowed or not.
SERVERS: Integer between 0 and 4. Specify the amount of server instances running on the machine. If not specified, the tool will detect if storpool_server is installed and how many instances are configured in the initialized disks (check with storpool_initdisk --list). If StorPool disks are not initialized the tool will exit with an error.
SET_CACHE_SIZE: Boolean, default is True. Specify whether the tool should calculate and set appropriate cache sizes for storpool_server instances in CACHEDIR. If set to 0 (False) the tool will oblige to the sizes set in storpool.conf and use them instead of calculating and setting them. Usable on server and hyperconverged setups.
SET_MEMSW: Boolean, default is True. Specify whether to set memory.memsw.limit in the cgconfig file.
SP_ALLOC_LIMIT: Non-negative integer, use one of the M or G suffixes. Specify the memory limit for the storpool/alloc slice. If not specified it will be calculated.
SP_COMMON_LIMIT: Non-negative integer, use one of the M or G suffixes. Specify the memory limit for the storpool/common slice. If not specified it will be calculated. On server nodes, the calculation requires StorPool disks to be initialized.
SP_CPUS_NOT_EXCLUSIVE: Boolean, default is 0 (False). Indicate that the StorPool CPUs are not exclusive. For more information, see Cpuset configuration.
SYSTEM_CPUS: List of CPUs for system and user slices. For details, see Cpuset isolation for the machine.slice.
SYSTEM_LIMIT: Non-negative integer, use one of the M or G suffixes; default is 2G. Specify the memory limit for the system slice.
USER_LIMIT: Non-negative integer, use one of the M or G suffixes; default is 2G. Specify the memory limit for the user slice; will be ignored on non-systemd machines.

Cpuset isolation for the machine.slice

Tip

This functionality is available starting with StorPool version 21.0 revision 21.0.576.6d56902e2.

The SYSTEM_CPUS parameter is set using the SYSTEM_CPUS=<list/range of cpus> format. The specified CPUs will be used for the system.slice and the user.slice of the machine. All non-system and non-storpool CPUs will be added to the machine.slice exclusively. Once set, the parameter will be saved in the cgtool.ini and loaded from it.

Note the following:

This is meant to be a one-time configuration to be used when setting up the machine for the first time, and only when there is a request to do this.
The parameter cannot be used when migrating.

Here is a simple example:

# storpool_cg conf system_cpus=0,16 converged=1 -N
...
socket:0
  core:0 cpus:[ 0 16]  --  system user      | system user
  core:1 cpus:[ 1 17]  --  storpool: server | storpool: server_1
  core:2 cpus:[ 2 18]  --  storpool: block  | storpool: bridge,beacon,rdma
  core:3 cpus:[ 3 19]  --  machine          | machine
  core:4 cpus:[ 4 20]  --  machine          | machine
  core:5 cpus:[ 5 21]  --  machine          | machine
  core:6 cpus:[ 6 22]  --  machine          | machine
  core:7 cpus:[ 7 23]  --  machine          | machine
socket:1
  core:0 cpus:[ 8 24]  --  machine          | machine
  core:1 cpus:[ 9 25]  --  machine          | machine
  core:2 cpus:[10 26]  --  machine          | machine
  core:3 cpus:[11 27]  --  machine          | machine
  core:4 cpus:[12 28]  --  machine          | machine
  core:5 cpus:[13 29]  --  machine          | machine
  core:6 cpus:[14 30]  --  machine          | machine
  core:7 cpus:[15 31]  --  machine          | machine
...
SYSTEM_CPUS=0,16 would have been saved

If you try to migrate this configuration while setting the system CPUs, it will fail:

# storpool_cg conf -M system_cpus=1,15
storpool_cg conf: error: cannot use --migrate with SYSTEM_CPUS

On machines that currently have machine.slice isolation, but the parameter is not set in the cgtool.ini file yet: storpool_cg should detect that, and automatically add it when you perform a migration:

# storpool_cg conf -M
Adding sniffed SYSTEM_CPUS=0,16
Loaded SYSTEM_CPUS=0,16
...
SYSTEM_CPUS=0,16 saved

Examples

Changing the system.slice and the user.slice

You could increase the system.slice on the machine, and then increase the user.slice as well:

# storpool_cg conf system_limit=8G
SYSTEM_LIMIT=8G saved

# storpool_cg conf user_limit=4G
Loaded SYSTEM_LIMIT=8G
USER_LIMIT=4G saved

# storpool_cg conf system_limit=6G
Loaded USER_LIMIT=4G
SYSTEM_LIMIT=6G saved

Resetting saved setting

You could reset the saved settings in the following way:

# storpool_cg conf --migrate user_limit=
Loaded SYSTEM_LIMIT=6G
Resetting USER_LIMIT
USER_LIMIT reset saved

In case storpool_cg fails at migrating the configuration the settings will not be saved:

# storpool_cg conf --migrate system_limit=
Resetting SYSTEM_LIMIT

# Migration checks
W: memory:system.slice new limit is less than max usage
E: if memory:system.slice is set to new limit, it will have more than 80% usage which is dangerous
W: Setup checks failed.

E: Operations exited unsuccessfully
SYSTEM_LIMIT reset NOT saved

To make the above example work:

# storpool_cg conf --migrate system_limit= --interactive
Resetting SYSTEM_LIMIT

# Migration checks
W: memory:system.slice new limit is less than max usage
E: if memory:system.slice is set to new limit, it will have more than 80% usage which is dangerous
Ignore that error? [y/N] yes
SYSTEM_LIMIT reset saved