Hugepages

StorPool uses the CPU support for bigger memory pages (hugepages). The number of hugepages needed by StorPool on a machine depends on the following:

  • The number of StorPool services the machine runs.

  • The number of NVMe disks and partitions used by StorPool.

Setting up hugepages

Hugepages can be reserved dynamically by the kernel by writing a desired number of pages to the /proc/sys/vm/nr_hugepages file. This value can be preserved across reboots by setting the variable vm.nr_hugepages in the /etc/sysctl.conf file or setting grub command line options.

However, on a NUMA machine, this will split the desired number of pages between all NUMA nodes. We do not want that and we want all hugepages for StorPool to be reserved on a single NUMA node. This can be done by writing the desired number of pages to the /sys/devices/system/node/node<N>/hugepages/hugepages-2048kB/nr_hugepages file where <N> is the NUMA node number. To keep that persistent across reboots a custom service for reservation is required.

The storpool_hugepages tool

The storpool_hugepages tool is designed to ease the process of setting up hugepages. It can calculate the number of hugepages needed for StorPool, and reserve them at the appropriate NUMA node.

Calculating StorPool hugepages needs at boot time can be dangerous because there are no guarantees that all block devices have been initialized by the kernel and are visible in the /dev filesystem. So the number of hugepages needed for NVMe devices is stored in the SP_NVME_HUGEPAGES variable in the /etc/stropool.conf.d/hugepages.conf file (see Using the files in /etc/storpool.conf.d/).

Running storpool_hugepages will allocate hugepages for all installed StorPool services and NVMe devices and will set the SP_NVME_HUGEPAGES variable.

At boot time the storpool_hugepages service will allocate hugepages for all installed StorPool services plus the number of pages for NVMe devices written in the /etc/storpool.conf.d/hugepages.conf file.

For new installations

The StorPool’s install.sh script will run storpool_hugepages every time it installs new services on the machine (including the initial installation). The only thing that needs to be done is to enable the storpool_hugepages service on the machine - you can do this with the storpool_ctl tool (see Managing services with storpool_ctl)

For installed machines

To configure hugepages on an already installed machine:

  1. Run storpool_hugepages once so that it allocates the appropriate number of hugepages and sets its configuration.

  2. Enable the storpool_hugepages service on the machine.

Hint

You can run storpool_hugepages -N -v to see for what services and NVMe devices are hugepages being allocated.