Creating cgroups configurations for hypervisors

Here you can find how to use the storpool_cg tool to configure freshly installed hypervisor machines.

Setting slice limits

If you think some of the slice limits should be different - for example, you want the system.slice limit to be 4G - you can do the following:

$ storpool_cg conf --noop system_limit=4G
W: NIC is expected to be on cpu 1
########## START SUMMARY ##########
slice: machine limit: 120872M
slice: storpool limit: 692M
  subslice: storpool/common limit: 692M
  subslice: storpool/alloc limit: 0G
slice: system limit: 4G
slice: user limit: 2G
###################################
cpus for StorPool: [1, 2, 21, 22]
socket:0
  core: 0 cpu: 0,20
  core: 1 cpu: 1,21 <--- 1 - nic; 21 - rdma
  core: 2 cpu: 2,22 <--- 2 - block; 22 - beacon
  core: 3 cpu: 3,23
  core: 4 cpu: 4,24
  core: 8 cpu: 5,25
  core: 9 cpu: 6,26
  core:10 cpu: 7,27
  core:11 cpu: 8,28
  core:12 cpu: 9,29
socket:1
  core: 0 cpu:10,30
  core: 1 cpu:11,31
  core: 2 cpu:12,32
  core: 3 cpu:13,33
  core: 4 cpu:14,34
  core: 8 cpu:15,35
  core: 9 cpu:16,36
  core:10 cpu:17,37
  core:11 cpu:18,38
  core:12 cpu:19,39
###################################

########### END SUMMARY ###########

In the same manner, you can pass machine_limit, user_limit sp_common_limit and sp_alloc_limit to the command line. Values in MB are also accepted with the M suffix.

Setting number of CPUs

If you want to dedicate more CPUs to StorPool, you can run storpool_cg with the cores=<N> parameter:

$ storpool_cg conf --noop cores=3
W: NIC is expected to be on cpu 1
########## START SUMMARY ##########
slice: machine limit: 122920M
slice: storpool limit: 692M
  subslice: storpool/common limit: 692M
  subslice: storpool/alloc limit: 0G
slice: system limit: 2G
slice: user limit: 2G
###################################
cpus for StorPool: [1, 2, 3, 21, 22, 23]
socket:0
  core: 0 cpu: 0,20
  core: 1 cpu: 1,21 <--- 1 - nic; 21 -
  core: 2 cpu: 2,22 <--- 2 - rdma; 22 -
  core: 3 cpu: 3,23 <--- 3 - block; 23 - beacon
  core: 4 cpu: 4,24
  core: 8 cpu: 5,25
  core: 9 cpu: 6,26
  core:10 cpu: 7,27
  core:11 cpu: 8,28
  core:12 cpu: 9,29
socket:1
  core: 0 cpu:10,30
  core: 1 cpu:11,31
  core: 2 cpu:12,32
  core: 3 cpu:13,33
  core: 4 cpu:14,34
  core: 8 cpu:15,35
  core: 9 cpu:16,36
  core:10 cpu:17,37
  core:11 cpu:18,38
  core:12 cpu:19,39
###################################

########### END SUMMARY ###########

Note that on hyper-threaded machines one core will add two CPUs, while on machines without (or with disabled) hyper-threading one core will add one CPU.

The storpool_cg tool detects which storpool services that need their own cpuset subslice are installed on the machine. It might happen (while unexpected) that you do not have all services installed yet.

Overriding services detection

You can override the services detection by specifying <service>=true or <service>=1. For example, to add a mgmt service to the above configuration:

$ storpool_cg conf --noop cores=3 mgmt=1
W: NIC is expected to be on cpu 1
########## START SUMMARY ##########
slice: machine limit: 120744M
slice: storpool limit: 2868M
  subslice: storpool/common limit: 692M
  subslice: storpool/alloc limit: 2176M
slice: system limit: 2G
slice: user limit: 2G
###################################
cpus for StorPool: [1, 2, 3, 21, 22, 23]
socket:0
  core: 0 cpu: 0,20
  core: 1 cpu: 1,21 <--- 1 - nic; 21 - rdma
  core: 2 cpu: 2,22 <--- 2 - block; 22 -
  core: 3 cpu: 3,23 <--- 3 - mgmt; 23 - beacon
  core: 4 cpu: 4,24
  core: 8 cpu: 5,25
  core: 9 cpu: 6,26
  core:10 cpu: 7,27
  core:11 cpu: 8,28
  core:12 cpu: 9,29
socket:1
  core: 0 cpu:10,30
  core: 1 cpu:11,31
  core: 2 cpu:12,32
  core: 3 cpu:13,33
  core: 4 cpu:14,34
  core: 8 cpu:15,35
  core: 9 cpu:16,36
  core:10 cpu:17,37
  core:11 cpu:18,38
  core:12 cpu:19,39
###################################

########### END SUMMARY ###########

The storpool_cg tool will also detect what driver the network card uses, and if it can be used with hardware acceleration in the current StorPool installation.

Overriding hardware acceleration

You can override the hardware acceleration detection by specifying iface_acc=true/false on the command line. Here is an example of the above configuration with hardware acceleration enabled:

$ storpool_cg conf --noop cores=3 mgmt=1 iface_acc=true
########## START SUMMARY ##########
slice: machine limit: 120744M
slice: storpool limit: 2868M
  subslice: storpool/common limit: 692M
  subslice: storpool/alloc limit: 2176M
slice: system limit: 2G
slice: user limit: 2G
###################################
cpus for StorPool: [1, 2, 3, 21, 22, 23]
socket:0
  core: 0 cpu: 0,20
  core: 1 cpu: 1,21 <--- 1 - rdma; 21 -
  core: 2 cpu: 2,22 <--- 2 - block; 22 -
  core: 3 cpu: 3,23 <--- 3 - mgmt; 23 - beacon
  core: 4 cpu: 4,24
  core: 8 cpu: 5,25
  core: 9 cpu: 6,26
  core:10 cpu: 7,27
  core:11 cpu: 8,28
  core:12 cpu: 9,29
socket:1
  core: 0 cpu:10,30
  core: 1 cpu:11,31
  core: 2 cpu:12,32
  core: 3 cpu:13,33
  core: 4 cpu:14,34
  core: 8 cpu:15,35
  core: 9 cpu:16,36
  core:10 cpu:17,37
  core:11 cpu:18,38
  core:12 cpu:19,39
###################################

########### END SUMMARY ###########

Note that storpool_cg will leave 1G memory for the kernel.

Setting memory for the kernel

If you want to change the amount of memory for the kernel, you can specify the kernel_mem=<X> command line parameter. For example, reserving 3G for the kernel:

$ storpool_cg conf --noop cores=3 mgmt=1 iface_acc=true kernel_mem=3G
########## START SUMMARY ##########
slice: machine limit: 118696M
slice: storpool limit: 2868M
  subslice: storpool/common limit: 692M
  subslice: storpool/alloc limit: 2176M
slice: system limit: 2G
slice: user limit: 2G
###################################
cpus for StorPool: [1, 2, 3, 21, 22, 23]
socket:0
  core: 0 cpu: 0,20
  core: 1 cpu: 1,21 <--- 1 - rdma; 21 -
  core: 2 cpu: 2,22 <--- 2 - block; 22 -
  core: 3 cpu: 3,23 <--- 3 - mgmt; 23 - beacon
  core: 4 cpu: 4,24
  core: 8 cpu: 5,25
  core: 9 cpu: 6,26
  core:10 cpu: 7,27
  core:11 cpu: 8,28
  core:12 cpu: 9,29
socket:1
  core: 0 cpu:10,30
  core: 1 cpu:11,31
  core: 2 cpu:12,32
  core: 3 cpu:13,33
  core: 4 cpu:14,34
  core: 8 cpu:15,35
  core: 9 cpu:16,36
  core:10 cpu:17,37
  core:11 cpu:18,38
  core:12 cpu:19,39
###################################

########### END SUMMARY ###########

Attention

storpool_cg will use CPUs for the storpool.slice from the local CPUs list of the StorPool network interfaces.