Hugepages
Memory hugepages and StorPool
StorPool uses the CPU support for bigger memory pages (hugepages). The number of hugepages needed by StorPool on a machine depends on the following:
The number of StorPool services the machine runs.
The number of NVMe disks and partitions used by StorPool.
Setting up hugepages
Hugepages can be reserved dynamically by the kernel by writing a desired number of pages to the /proc/sys/vm/nr_hugepages
file. This value can be preserved across reboots by setting the variable vm.nr_hugepages
in the /etc/sysctl.conf
file or setting grub command line options.
However, on a NUMA machine, this will split the desired number of pages between all NUMA nodes. We do not want that and we want all hugepages for StorPool to be reserved on a single NUMA node. This can be done by writing the desired number of pages to the /sys/devices/system/node/node<N>/hugepages/hugepages-2048kB/nr_hugepages
file where <N> is the NUMA node number. To keep that persistent across reboots a custom service for reservation is required.
storpool_hugepages
storpool_hugepages
is designed to ease the process of setting up hugepages.
It can calculate the number of hugepages needed for StorPool, and reserve them
at the appropriate NUMA node.
Calculating StorPool hugepages needs at boot time can be dangerous because there
are no guarantees that all block devices have been initialized by the kernel and
are visible in the /dev
filesystem. So the number of hugepages needed for
NVMe devices is stored in the SP_NVME_HUGEPAGES
variable in the
stropool.conf.d/hugepages.conf
file.
Running storpool_hugepages
will allocate hugepages for all installed
StorPool services and NVMe devices and will set the SP_NVME_HUGEPAGES
variable in the config.
At boot time the storpool_hugepages
service will allocate hugepages for all
installed StorPool services plus the number of pages for NVMe devices
written in the storpool.conf.d/hugepages.conf
file.
For new installations
The StorPool’s install.sh
script will run storpool_hugepages
every time
it installs new services on the machine (including the initial installation).
The only thing that needs to be done is to enable the storpool_hugepages
service on the machine.
For installed machines
To configure hugepages on an already installed machine:
Run
storpool_hugepages
once so that it allocates the appropriate number of hugepages and sets its configuration.Enable the
storpool_hugepages
service on the machine.
Hint
You can run storpool_hugepages -N -v
to see for what services and
NVMe devices are hugepages being allocated.