Updating already configured machines

Sometimes you might need to update machines that are already configured and run the StorPool services. A possible solution for this scenario is to create a new configuration and reboot the machine. However, often you won’t be able to reboot; for these cases, storpool_cg offers an option to try to migrate configuration changes live without the need for a reboot. It is activated by the -M (--migrate) command line option.

When you have a configuration that you want to apply to a machine (you know with what options you want to run storpool_cg) you have two options:

  • Run the storpool_cg to create cgconfig.d files, and then reboot.

  • Use storpool_cg conf with the same options plus the -M option, and let it try to apply it.

Attention

Note the following:

  • Before attempting a live migration, storpool_cg will run a series of checks to verify that it is safe to try the migration.

  • Migrating machines with storpool_cg is a pseudo-transactional procedure. If the migration process fails, a rollback procedure will be attempted to restore the initial machine state. The rollback operation is not guaranteed to succeed! Some ‘extreme’ conditions must have occurred for the rollback to fail.

You can use the migration in the following cases:

  • Changing slice limits

  • Enabling hardware acceleration

  • Adding and removing services

  • Adding and removing disks

Migrating to new-style configuration

Let’s look at the following machine:

$ storpool_cg print
slice: storpool.slice limit: 26G
slice: system.slice limit: 2G
slice: user.slice limit: 2G
socket:0
  core:0 cpus:[ 0  1]  --  nic      | rdma
  core:1 cpus:[ 2  3]  --  server   | block
  core:2 cpus:[ 4  5]  --  server_1 |
  core:3 cpus:[ 6  7]  --  server_2 | beacon
  core:4 cpus:[ 8  9]  --
  core:5 cpus:[10 11]  --
  core:6 cpus:[12 13]  --
  core:7 cpus:[14 15]  --

You can run a ‘dry run’ migration with -NM to see the desired configuration and the steps the tool will take to achieve it. All other arguments of storpool_cg conf can be used with -M, so (for example) if you need to tweak the number of cores, you can still use cores=4.

$ storpool_cg conf -NM
W: NIC is expected to be on cpu 2
########## START SUMMARY ##########
slice: storpool limit: 26696M
  subslice: storpool/common limit: 26696M
  subslice: storpool/alloc limit: 0G
slice: system limit: 2G
slice: user limit: 2G
###################################
cpus for StorPool: [2, 3, 4, 5, 6, 7]
socket:0
  core: 0 cpu: 0, 1
  core: 1 cpu: 2, 3 <--- 2 - nic; 3 - rdma
  core: 2 cpu: 4, 5 <--- 4 - server; 5 - server_1
  core: 3 cpu: 6, 7 <--- 6 - server_2; 7 - block,beacon
  core: 4 cpu: 8, 9
  core: 5 cpu:10,11
  core: 6 cpu:12,13
  core: 7 cpu:14,15
########### END SUMMARY ###########
echo 2 > /sys/fs/cgroup/cpuset/storpool.slice/rdma/cpuset.cpus
echo 3 > /sys/fs/cgroup/cpuset/storpool.slice/server/cpuset.cpus
echo 4 > /sys/fs/cgroup/cpuset/storpool.slice/block/cpuset.cpus
echo 5 > /sys/fs/cgroup/cpuset/storpool.slice/server_1/cpuset.cpus
echo 2-7 > /sys/fs/cgroup/cpuset/storpool.slice/cpuset.cpus
echo 0-1,8-15 > /sys/fs/cgroup/cpuset/user.slice/cpuset.cpus
echo 0-1,8-15 > /sys/fs/cgroup/cpuset/system.slice/cpuset.cpus
echo 4 > /sys/fs/cgroup/cpuset/storpool.slice/server/cpuset.cpus
echo 3 > /sys/fs/cgroup/cpuset/storpool.slice/rdma/cpuset.cpus
echo 7 > /sys/fs/cgroup/cpuset/storpool.slice/block/cpuset.cpus
echo 26696M > /sys/fs/cgroup/memory/storpool.slice/memory.memsw.limit_in_bytes
echo 26696M > /sys/fs/cgroup/memory/storpool.slice/memory.limit_in_bytes
mkdir /sys/fs/cgroup/memory/storpool.slice/common
echo 1 > /sys/fs/cgroup/memory/storpool.slice/common/memory.use_hierarchy
echo 3 > /sys/fs/cgroup/memory/storpool.slice/common/memory.move_charge_at_immigrate
echo 26696M > /sys/fs/cgroup/memory/storpool.slice/common/memory.limit_in_bytes
mkdir /sys/fs/cgroup/memory/storpool.slice/alloc
echo 1 > /sys/fs/cgroup/memory/storpool.slice/alloc/memory.use_hierarchy
echo 3 > /sys/fs/cgroup/memory/storpool.slice/alloc/memory.move_charge_at_immigrate
echo 0G > /sys/fs/cgroup/memory/storpool.slice/alloc/memory.limit_in_bytes
echo 6143 > /sys/fs/cgroup/memory/storpool.slice/common/cgroup.procs
echo 6682 > /sys/fs/cgroup/memory/storpool.slice/common/cgroup.procs
echo 6692 > /sys/fs/cgroup/memory/storpool.slice/common/cgroup.procs
echo 6913 > /sys/fs/cgroup/memory/storpool.slice/common/cgroup.procs
echo 6926 > /sys/fs/cgroup/memory/storpool.slice/common/cgroup.procs
echo 6977 > /sys/fs/cgroup/memory/storpool.slice/common/cgroup.procs
echo 6987 > /sys/fs/cgroup/memory/storpool.slice/common/cgroup.procs
echo 7174 > /sys/fs/cgroup/memory/storpool.slice/common/cgroup.procs
echo 7185 > /sys/fs/cgroup/memory/storpool.slice/common/cgroup.procs
echo 7585 > /sys/fs/cgroup/memory/storpool.slice/common/cgroup.procs
echo 7604 > /sys/fs/cgroup/memory/storpool.slice/common/cgroup.procs

When you are happy with the configuration and want to migrate to it, run the migration without -N; note that if the new configuration is taking more CPUs or memory for StorPool, you need to add the -E / --expand option (see Configuration options and parameters). Then check if everything is OK with storpool_cg print -S and storpool_cg check.

$ storpool_cg conf --migrate
W: NIC is expected to be on cpu 2

$ storpool_cg print --slices
slice: storpool.slice limit: 26696M
  subslice: storpool.slice/alloc limit: 0G
  subslice: storpool.slice/common limit: 26696M
slice: system.slice limit: 2G
slice: user.slice limit: 2G
socket:0
  core:0 cpus:[ 0  1]  --  system user        | system user
  core:1 cpus:[ 2  3]  --  storpool: nic      | storpool: rdma
  core:2 cpus:[ 4  5]  --  storpool: server   | storpool: server_1
  core:3 cpus:[ 6  7]  --  storpool: server_2 | storpool: beacon,block
  core:4 cpus:[ 8  9]  --  system user        | system user
  core:5 cpus:[10 11]  --  system user        | system user
  core:6 cpus:[12 13]  --  system user        | system user
  core:7 cpus:[14 15]  --  system user        | system user

$ storpool_cg check
M: ==== memory ====
W: memory:system.slice has more than 80% usage

Attention

One more thing you need to check after a migration is the output of storpool_process reclassify -N -v. If it suggests moving some processes to different cgroups than the ones they are currently running in, you can do it by running just storpool_process reclassify. Note that storpool_process will make suggestions based on the current SP_{X}_CGROUPS variables in the storpool.conf.

Migrating machines with previous manual setting of SYSTEM_LIMIT or USER_LIMIT

Starting with StorPool version 21.0 revision 21.0.576.6d56902e2 the default system.slice limit is 4G. This could create situations where a migration will not be possible, because of the reduced limit. In such situation the storpool_cg tool will fail and ask you to add a sensible value for the system.slice limit:

# storpool_cg conf -M
E: Current system.slice memory usage 4450M is > the default limit 4G. Please specify some sensible value for SYSTEM_LIMIT

When migrating, the tool will check if the system and user limits are different than the default (and also if the values are whole numbers in GiB), and will ask you if you want to load them:

# storpool_cg conf -M
Sniffed USER_LIMIT=4G, do you want to load it? [y/N] y
Adding sniffed USER_LIMIT=4G
Sniffed SYSTEM_LIMIT=6G, do you want to load it? [y/N] y
Adding sniffed SYSTEM_LIMIT=6G
Loaded USER_LIMIT=4G
Loaded SYSTEM_LIMIT=6G
...
USER_LIMIT=4G saved
SYSTEM_LIMIT=6G saved