StorPool services
A StorPool installation consists of services that take care of different functionality on each node participating in the cluster:
For details about how to control the services, see Managing services with storpool_ctl.
storpool_beacon
The beacon must be the first StorPool process started on all nodes in the cluster. It
informs all members about the availability of the node on which it is installed.
If the number of the visible nodes changes, every storpool_beacon service
checks that its node still participates is the quorum - which means it can
communicate with more than half of the expected nodes, including itself (see
SP_EXPECTED_NODES in Expected nodes).
If the storpool_beacon service starts successfully, it will send to the
system log (/var/log/messages, /var/log/syslog, or similar) messages as
those shown below for every node that comes up in the StorPool cluster:
[snip]
Jan 21 16:22:18 s01 storpool_beacon[18839]: [info] incVotes(1) from 0 to 1, voteOwner 1
Jan 21 16:23:10 s01 storpool_beacon[18839]: [info] peer 2, beaconStatus UP bootupTime 1390314187662389
Jan 21 16:23:10 s01 storpool_beacon[18839]: [info] incVotes(1) from 1 to 2, voteOwner 2
Jan 21 16:23:10 s01 storpool_beacon[18839]: [info] peer up 1
[snip]
storpool_server
The storpool_server service must be started on each node that provides its
storage devices (HDD, SSD, or NVMe drives) to the cluster. If the service starts
successfully, all the drives intended to be used as StorPool disks should be
listed in the system log, as shown in the example below:
Dec 14 09:54:19 s11 storpool_server[13658]: [info] /dev/sdl1: adding as data disk 1101 (ssd)
Dec 14 09:54:19 s11 storpool_server[13658]: [info] /dev/sdb1: adding as data disk 1111
Dec 14 09:54:20 s11 storpool_server[13658]: [info] /dev/sda1: adding as data disk 1114
Dec 14 09:54:20 s11 storpool_server[13658]: [info] /dev/sdk1: adding as data disk 1102 (ssd)
Dec 14 09:54:20 s11 storpool_server[13658]: [info] /dev/sdj1: adding as data disk 1113
Dec 14 09:54:22 s11 storpool_server[13658]: [info] /dev/sdi1: adding as data disk 1112
On a dedicated node (for example, one with a larger amount of spare resources)
you can start more than one instances of storpool_server service (up to
seven); for details, see Multi-server.
The server services are responsible for the following tasks:
Handling writes, reads, and data integrity checksums during normal conditions
Re-testing and re-adding storage devices ejected by mistake or by automated checks.
Executing background tasks like rebalancing, rebuilding/self-healing, and scrubbing.
You can manage the storpool_server service using the command-line interface, for details see Server.
storpool_block
The storpool_block service provides the client (initiator) functionality.
StorPool volumes can be attached only to the nodes where this service is
running. When attached to a node, a volume can be used and manipulated as a
regular block device via the /dev/stopool/{volume_name} symlink:
# lsblk /dev/storpool/test
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sp-2 251:2 0 100G 0 disk
Thanks to this mechanism, each volume appears to KVM hypervisors and bare-metal Linux hosts like a local storage device.
In the write direction, the block client knows which drives each incoming data block should be written to and chooses the server services that should handle each operation. It generates a data integrity checksum for every data block. The user data, metadata, and checksums are sent over the network and stored together, ensuring data integrity. After the server service commits the data to a storage device, it acknowledges the completion of each operation, and the block service confirms the user write operation.
In the read direction, the block client requests data directly from the right server service (avoiding unnecessary network hops), validates that data integrity checksums match the expected values, and completes the user read operation. If the checksums do not match, the block client retries the operations with another copy of the data. If the data in one copy has been corrupted, the storpool_server service schedules the recovery from a copy where the checksums match the expected values.
The block client is controlled and monitored through the cluster-level StorPool API for attachments, renames, resizing, and other functions. It also implements purpose-built buffer management, congestion avoidance, repeated transmission, and other algorithms that ensure the extreme speed and reliability of StorPool storage systems.
You can manage the service using the command-line interface, for details see Client.
storpool_mgmt
The storpool_mgmt service should be started on at least two nodes in the
cluster. It is primarily responsible for listening for incoming HTTP
REST API calls and authenticating and executing them within the
storage cluster. Commands from the command line interface (see
CLI reference) and graphical user interface (see Graphical user interface) are
converted to API calls and executed by the active management service.
Thanks to the management service, each StorPool storage cluster behaves as an integrated entity, and configuration and state are always synchronized and consistent across all nodes. This allows you to:
Automate storage-related functionalities when working with platforms supported by Storpool (see StorPool integrations), and also with custom-built cloud management solutions
Build self-service user interfaces for end-users to self-provision virtual machines and change workload settings on the fly
Generate automatic monthly billing for capacity consumed by end-users
Build tools that provide end-users with usage reporting for their environments
Automate migrations between virtualization technologies and storage products
An automatic failover mechanism is available: when the node with the
active storpool_mgmt service fails, the SP_API_HTTP_HOST IP address is
automatically configured on the next node with the lowest SP_OURID with a
running storpool_mgmt service; for details about these options, see
Address for API management and Identification and voting.
storpool_bridge
The storpool_bridge service is started on two or more nodes in the cluster,
with one being active (similarly to the storpool_mgmt service). This service
synchronizes snapshots for the backup and disaster recovery use cases between
the current cluster and one or more StorPool clusters in different locations.
For details, see Multi-site and multi-cluster.
storpool_controller
The storpool_controller service is started on all nodes running the
storpool_server service. It collects information from all
storpool_server instances in order to provide statistics data to the API.
Note
The storpool_controller service requires port 47567 to be open on
the nodes where the API (storpool_mgmt) service is running.
storpool_nvmed
The storpool_nvmed service is started on all nodes that have the
storpool_server service. It handles the management of the NVMe devices,
their unplugging from kernel’s NVMe driver, and passing to the storpool_pci
or vfio_pci drivers. You can configure this using the
SP_NVME_PCI_DRIVER option in the /etc/storpool.conf file. For more
information, see NVMe SSD drives.
storpool_stat
The storpool_stat service is started on all nodes. It collects the following
system metrics from all nodes:
CPU stats - queue run/wait, user, system, and so on, per CPU
Memory usage stats per cgroup
Network stats for the StorPool services
The I/O stats of the system drives
Per-host validating service checks (for example, if there are processes in the root cgroup, the API is reachable if configured, and so on)
On some nodes it collects additional information:
On all nodes with the
storpool_blockservice: the I/O stats of all attached StorPool volumes;On server nodes: stats for the operations of
storpool_serverwith the drives.
For more information, see Monitoring metrics collected.
The collected data can be viewed at analytics.storpool.com. It can also
be submitted to an InfluxDB instance run by your organization; this can be
configured in storpool.conf, for details see Monitoring and issue reports.
storpool_qos
The storpool_qos service tracks changes for volumes that match certain
criteria, and takes care for updating the I/O performance settings of the
matching volumes. For details, see Quality of service.
storpool_iscsi
storpool_iscsi is a client service (like storpool_block) that translates
the operations received via iSCSI to the StorPool internal protocol. This
service is particularly useful in environments where the native block protocol
of StorPool is not supported (for example, VMware or Microsoft Hyper-V), or in
complex setups like OpenStack deployed via Juju charms. It provides
administrative separation, allowing for a more scalable and flexible approach to
client management and supporting more than 64 clients per storage cluster.
In environments using the iSCSI block storage protocol, storpool_iscsi is
installed on all storage and hyper-converged nodes.
The service differs from the rest of the StorPool services in that it requires 4
network interfaces instead of the regular 2. Two of them are used for
communication with the cluster, and the other two are used for providing iSCSI
service to initiators.
The service itself needs a separate IP address for every portal and network (different than the ones used in the host kernel). These addresses are handled by the service’s TCP/IP stack and have their own MAC addresses.
Note
Currently, re-use of the host IP address for the iSCSI service is not possible.
Note that the iSCSI service cannot operate without hardware acceleration. For details, see Network interfaces and the StorPool System Requirements document.
For more information on configuring and using the service, see iSCSI options and Setting iSCSI targets
storpool_abrtsync
The storpool_abrtsync service automatically sends reports about aborted
services to the monitoring system used by StorPool; see Monitoring alerts.
storpool_havm
The storpool_havm (highly available virtual machine tracking) service tracks
the state of one or more virtual machines and keeps them active on one of the
nodes in the cluster. The sole purpose of this service is to offload the
orchestration responsibility for virtual machines where the fast startup after a
failover event is crucial.
A virtual machine is configured on all nodes where the storpool API
(storpool_mgmt service) is running with a predefined libvirt XML and
predefined volume names. The storpool_havm@<vm_name> service gets enabled
on each API node in the cluster, then starts tracking the state of this virtual
machine.
The VM is kept active on the active API node. In the typical case where the
active API changes due to service restart, the VM gets live-migrated to the new
active API node.
In case of a failure of the node where the active API was last running, the service takes care to fence the block devices on the old API node and to start the VM on the present active node.
The primary use case is virtual machines for NFS or OpenNebula front-end.
This service is available starting with release 20.0 revision 20.0.19.1a208ffab.
storpool_logd
The StorPool log daemon (storpool_logd) receives log messages from all
StorPool services working in a cluster and the Linux kernel logs for further
analysis and advanced monitoring.
Tracking the storage service logs for the whole cluster enables more advanced monitoring, as well as safer maintenance operations. In the long term, it allows for:
Better accountability
Reduced times for investigating issues or incidents
Logs inspection for longer periods
Retroactive detection of issues identified in a production cluster in the whole installed base
The service reads messages from two log streams, enqueues them into a persistent backend, and sends them to StorPool. Once the reception is confirmed, messages are removed from the backend.
The service tries its best to ensure the logs are delivered. Logs can survive
between process and entire node restarts. storpool_logd prioritizes
persisted messages over incoming ones, so new messages will be dropped if the
persistent storage is full.
Monitoring cluster-wide logs allows raising alerts on cluster-wide events that are based either on specific messages or classified message frequency thresholds. The ultimate goal is to lower the risk of accidents and unplanned downtime by proactively detecting issues based on similarities in the observed services behavior.
The main use cases for the service are:
Proactively detect abnormal and unexpected messages and raise alerts.
Proactively detect an abnormal rate of messages and raise alerts
Enhance situational awareness by allowing operators to monitor the logs for a whole cluster in one view
Allow for easier tracking of newly discovered failure scenarios
There are relevant configuration sections for the service if a proxy is required to be able to send data, or if a custom instance is used to override the URL of the default instance used by StorPool. For details, see Configuring the StorPool log daemon service.
This service is available starting with release 20.0 revision 20.0.372.4cd1679db.
storpool_gui
The storpool_gui service is installed on every node where storpool_mgmt is installed.
Having this service running on StorPool cluster allows users to perform some of the most important operations via a browser, instead of using the CLI (see CLI reference) or the REST API.
For details on how to access and use the Web interface, see Graphical user interface.
The configuration options for the service – like SP_GUI_IFACE, SP_GUI_HTTP_HOST, and SP_GUI_HTTP_PORT – are set in the storpool.conf file; for details, see Host and port for the Web interface.
storpool_volumecare
This service can be used when VolumeCare is installed.