Hugepages

Memory hugepages and StorPool

StorPool uses the CPU support for bigger memory pages (hugepages). The number of hugepages needed by StorPool on a machine depends on the following:

  • The number of StorPool services the machine runs.

  • The number of NVMe disks and partitions used by StorPool.

Setting up hugepages

Hugepages can be reserved dynamically by the kernel by writing a desired number of pages to the /proc/sys/vm/nr_hugepages file. This value can be preserved across reboots by setting the variable vm.nr_hugepages in the /etc/sysctl.conf file or setting grub command line options.

However, on a NUMA machine, this will split the desired number of pages between all NUMA nodes. We do not want that and we want all hugepages for StorPool to be reserved on a single NUMA node. This can be done by writing the desired number of pages to the /sys/devices/system/node/node<N>/hugepages/hugepages-2048kB/nr_hugepages file where <N> is the NUMA node number. To keep that persistent across reboots a custom service for reservation is required.

storpool_hugepages

storpool_hugepages is designed to ease the process of setting up hugepages. It can calculate the number of hugepages needed for StorPool, and reserve them at the appropriate NUMA node.

Calculating StorPool hugepages needs at boot time can be dangerous because there are no guarantees that all block devices have been initialized by the kernel and are visible in the /dev filesystem. So the number of hugepages needed for NVMe devices is stored in the SP_NVME_HUGEPAGES variable in the stropool.conf.d/hugepages.conf file.

Running storpool_hugepages will allocate hugepages for all installed StorPool services and NVMe devices and will set the SP_NVME_HUGEPAGES variable in the config.

At boot time the storpool_hugepages service will allocate hugepages for all installed StorPool services plus the number of pages for NVMe devices written in the storpool.conf.d/hugepages.conf file.

For new installations

The StorPool’s install.sh script will run storpool_hugepages every time it installs new services on the machine (including the initial installation). The only thing that needs to be done is to enable the storpool_hugepages service on the machine.

For installed machines

To configure hugepages on an already installed machine:

  1. Run storpool_hugepages once so that it allocates the appropriate number of hugepages and sets its configuration.

  2. Enable the storpool_hugepages service on the machine.

Hint

You can run storpool_hugepages -N -v to see for what services and NVMe devices are hugepages being allocated.