Adding and removing nodes

The servers participating in a StorPool cluster are referred to as nodes. Based on the services run on a node (see Background services), it can be used as a storage node, a client node, or both. The most common tasks when administering a cluster are adding, removing, and recovering nodes.

Prerequisites

Adding nodes

This procedure applies for cases where your cluster meets the following conditions (before adding new nodes):

  • It has two or more voting nodes.

  • Their number is greater than the number of new nodes you are adding.

Usually all server nodes are voting, and client-only nodes are non-voting.

Prerequisites

To add a new node to the cluster, first ensure all of the following conditions are met:

  • StorPool is installed on the new node and the StorPool services are stopped. For details, see Installation and setup and Managing services with storpool_ctl.

  • Make sure all nodes are up and in the cluster! You can check this by running the storpool net list command on one of the available nodes; see Network for details.

Procedure

Update the /etc/storpool.conf file as follows:

  1. Only when adding a new voting node: on the new node, set the value of the SP_EXPECTED_NODES option so that it is equal to the new number of voting nodes (with the new node added). For more information, see Expected nodes.

  2. Add a section for the new node, with node-specific settings.

  3. Copy the updated storpool.conf file from the new node to all other nodes in the cluster. Note the following:

    • There is no need to restart the services on the existing nodes after this change.

    • It is important that each node has valid configuration sections for all nodes in the cluster in its local /etc/storpool.conf file. Keep consistent /etc/storpool.conf files across all nodes in the cluster.

  4. Start the StorPool services on the new node.

Removing nodes

When removing a node, first eject all the drives on this node (to stop using them), and then update the configuration of the rest of the nodes. Note the following:

  • If the monitoring of your cluster is performed by StorPool (see Monitoring alerts): notify StorPool support that a node will be decommissioned and monitoring and notifications shall be stopped for this node.

  • In case you plan returning the node after removing it, consider using maintenance mode. For details, see Maintenance mode.

Stopping drive usage

Log on to the node you want to remove and perform the following steps:

  1. If this is a server node, follow the Removing a drive without replacement procedure.

  2. Check quorum. If the node you are planning to remove is configured as voting, verify that there will be enough voting nodes remaining in the cluster after this node is disconnected.

    Check the expected and actual number of voting nodes in the cluster with:

    storpool net list
...
Quorum status: 4 voting beacons up out of 4 expected

    Attention

    After the node is removed, the number of remaining voting nodes shall be more than 50% of the expected nodes listed above. If this requirement is not satisfied, decrease the number of expected nodes before disconnecting the node (see below).

  3. After re-balancing is completed, disconnect the node from the cluster by running the following commands:

    storpool_ctl disable
storpool_ctl stop

    For details, see Managing services with storpool_ctl.

  4. Check that the node is not part of the cluster anymore with the storpool service list command. There should be no services listed for the node you removed (check it’s ID is not present in the results).

The node can now be physically disconnected from the storage network.

Updating cluster configuration

After the node is removed, perform the following steps:

  1. Update the main configuration file on all remaining nodes in the cluster:

    1. Log on to one of the available nodes and start editing the /etc/storpool.conf file.

    2. If the removed node was a voting one, decrease the value of the SP_EXPECTED_NODES option to be equal to the actual number of voting nodes remaining in the cluster.

    3. Remove the node-specific section for the removed node.

    4. Save the file.

    5. Copy the updated storpool.conf file to all other nodes in the cluster.

      Note that there is no need to restart any services after this change.

  2. If the number of expected voting nodes in the cluster was reduced:

    1. Log on to the current active management node.

      You can find it using the storpool service list command and checking the node with active storpool_mgmt service.

    2. Use the storpool net list command to check the quorum.

      If the currently connected nodes do not have a quorum, bring up enough nodes to achieve a quorum.

    3. Make the new value of expected voting nodes active:

      echo "expected ${EXPECTED}" | socat - unix-sendto:/var/run/storpool/beacon.cmd.sock

      Here ${EXPECTED} is the new number of expected voting nodes. For example, if this value is 4, the command would look like this: echo "expected 4" | socat - unix-sendto:/var/run/storpool/beacon.cmd.sock.

    4. Check the expected number if voting nodes is correct with:

      storpool net list | grep 'expected'

  3. Set the StorPool cluster to forget all drives previously used by the removed node.

    For each drive of this node, run the following command: storpool disk <diskID> forget

Recovering nodes

It might happen that due to software or hardware issues a node stops participating in the cluster. Here are a few examples:

  • The storpool_server service is not available.

  • Incorrect host OS configuration.

  • A reboot of the node for a kernel or a package upgrade (that requires reboot), and no kernel modules were installed for the new kernel.

  • A service (like storpool_server) was not configured to start when the node boots.

  • Network interface issues.

  • Drive or controller issues.

In such situations, you can try bringing the node back to the cluster as described in Degraded state.