StorPool iSCSI balance procedure

Introduction

Since the introduction of iSCSI scalable, we have a list of controllers for each target. The target is always served from the first controller in the list that’s up (i.e. the portalGroup redirects the connections to that one, and that’s the only controller that can serve the target/volume). If the first controller that’s up in the list is changed, the session is disconnected to connect to the new first one in the list.

The iSCSI balancing is helpful when the controllers have an uneven number of targets potentially causing performance issues, or when new iSCSI controller nodes are added or old ones removed from the cluster. Manually adding a number of controllers to each target’s controller list is a difficult and confusing task. For this case, we can use the iscsi_balancer tool. Here are explanatory instructions on how this tool works.

The tool is part of the StorPool management package and is installed as /usr/lib/storpool/iscsi_balancer on the management nodes:

# /usr/lib/storpool/iscsi_balancer --help
usage: iscsi_balancer [-h] [--json] {show,balance,state} ...

StorPool iSCSI Balancer — inspect and rebalance iSCSI targets across
controllers.

options:
  -h, --help            show this help message and exit
  --json                output in JSON format (for scripting)

commands:
  {show,balance,state}
    show                display target distribution (or 'show load' for
                        metrics)
    balance             plan or execute a target rebalance
    state               save or restore target controller state

Several behaviors apply to all balance strategies:

  • All balance commands are a dry run by default - they only print the planned changes. Add --exec to apply them.

  • Before applying any change, the tool automatically saves the current state of all targets to a JSON file in the working directory, which can later be restored with state restore (see Saving and restoring state).

  • For the strategies that cause session reconnects (intrusive, load, and move), the tool automatically verifies that all iSCSI sessions reconnect after the change (see Automatic session verification).

  • Adding --commands prints the equivalent storpool_q commands instead of executing anything.

  • The global --json flag switches any command’s output to JSON, which is convenient for scripting.

Note

The target’s controller list can contain a maximum of 7 controllers.

Checking the current target distribution

We can check how the targets are currently distributed across the controllers with the show command:

# /usr/lib/storpool/iscsi_balancer show

Targets: 6 across 3 controller(s)

Controller  Targets
----------  -------
         1        1
         2        2
         3        3

Target                               Controller  Backup Order
-----------------------------------  ----------  ------------
iqn.2019-08.com.example:web-vol1              1  [1, 2, 3]
iqn.2019-08.com.example:web-vol2              2  [2, 3, 1]
iqn.2019-08.com.example:db-vol1               2  [2, 1, 3]
iqn.2019-08.com.example:db-vol2               3  [3, 2, 1]
iqn.2019-08.com.example:backup-vol1           3  [3, 2, 1]
iqn.2019-08.com.example:backup-vol2           3  [3, 1, 2]

The first table shows how many targets are served by each controller; the second one shows each target’s current controller and its full controller (failover) list.

Checking the load per controller

Balancing is required not only for the distribution of targets, but also for the SCSI load between the target and the initiator. The load of a session is the average number of SCSI commands the initiator has sent per second since the session started. Each read and each write counts as one command. The size of the data transferred is separate from this number.

The average covers the whole life of the session. A session that was busy earlier can still show a high load, and a session that has just become busy can still show a low one.

Here is the same calculation with jq:

# storpool -j iscsi sessions list | jq -r '.data.sessions[] | select(.stats) | .target + " " + ((.stats.scsi/(now-.timeCreated)) | tostring)'

The show load command computes this for every active session. It adds the sessions of each volume together, then adds those volumes into the controller total. The table shows whole numbers, with the heaviest controller first:

# /usr/lib/storpool/iscsi_balancer show load

Controller 1:
  Volume  Load
--------  ----
web-vol1  5580

Controller 2:
  Volume  Load
--------  ----
web-vol2  1511
db-vol1    507

Controller 3:
  Volume     Load
-----------  ----
db-vol2      1130
backup-vol1   590
backup-vol2    41

Controller  Total Load
----------  ----------
         1        5580
         2        2018
         3        1761

To show only the targets of a single controller, pass the controller ID:

# /usr/lib/storpool/iscsi_balancer show load 3

Controller 3:
  Volume     Load
-----------  ----
db-vol2      1130
backup-vol1   590
backup-vol2    41

Controller  Total Load
----------  ----------
         1        5580
         2        2018
         3        1761

Balancing strategies

There are four balancing strategies:

# /usr/lib/storpool/iscsi_balancer balance --help
usage: iscsi_balancer balance [-h] {backup,intrusive,load,move} ...

options:
  -h, --help            show this help message and exit

strategies:
  {backup,intrusive,load,move}
    backup              reorder failover controllers (no reconnects)
    intrusive           redistribute targets across controllers (causes
                        reconnects!)
    load                move heaviest target from most-loaded to least-loaded
                        controller
    move                move a specific volume to a controller

  • backup is non-intrusive - it keeps each target’s current controller and only re-arranges the failover (backup) controllers in the list, so no sessions are disconnected.

  • intrusive, load, and move may change the current controller of a target and therefore cause iSCSI session reconnects.

We can determine which strategy is necessary based on the current target distribution and load per controller shown above.

Backup balancing (non-intrusive)

This strategy keeps the current controller of each target in place and only re-arranges the rest of the controller list, so that the failover order is spread evenly across the remaining controllers. No iSCSI sessions are disconnected.

Print the plan (dry run):

# /usr/lib/storpool/iscsi_balancer balance backup

Strategy: backup (reorder failover controllers, no reconnects)
Planned changes: 6 target(s)

Volume       Controller     Current Order  New Order
-----------  -------------  -------------  ---------
web-vol1     1 (no change)  [1, 2, 3]      [1, 3, 2]
web-vol2     2 (no change)  [2, 3, 1]      [2, 1, 3]
db-vol1      2 (no change)  [2, 1, 3]      [2, 3, 1]
db-vol2      3 (no change)  [3, 2, 1]      [3, 1, 2]
backup-vol1  3 (no change)  [3, 2, 1]      [3, 1, 2]
backup-vol2  3 (no change)  [3, 1, 2]      [3, 2, 1]

Run with --exec to apply. State will be auto-saved before execution.

Execute the balancing:

# /usr/lib/storpool/iscsi_balancer balance backup --exec

Intrusive balancing

This strategy fully redistributes the targets across all controllers, so that each controller serves an even number of targets.

Attention

This operation can change the current controller of a target and will lead to iSCSI session reconnects.

Print the plan (dry run):

# /usr/lib/storpool/iscsi_balancer balance intrusive

Strategy: intrusive (will cause iSCSI session reconnects!)
Planned changes: 6 target(s)

Volume       Controller     Current Order  New Order
-----------  -------------  -------------  ---------
web-vol1     1 (no change)  [1, 2, 3]      [1, 2, 3]
web-vol2     2 (no change)  [2, 3, 1]      [2, 3, 1]
db-vol1      2 -> 3         [2, 1, 3]      [3, 1, 2]
db-vol2      3 -> 1         [3, 2, 1]      [1, 3, 2]
backup-vol1  3 -> 2         [3, 2, 1]      [2, 1, 3]
backup-vol2  3 (no change)  [3, 1, 2]      [3, 2, 1]

Run with --exec to apply. State will be auto-saved before execution.

Execute the balancing:

# /usr/lib/storpool/iscsi_balancer balance intrusive --exec

Balancing by load

This strategy takes the busiest volume of the most loaded controller and moves it to the least loaded controller (one move per execution), thus improving the overall performance of the cluster by removing excessive load from one of the controllers and spreading it to the others. Run it repeatedly to perform multiple moves, checking the result with show load in between.

Controllers that serve a single volume are skipped - moving their only target would merely shift the same load to another controller without actually rebalancing anything. In the example below controller 1 is the most loaded one, but it serves only web-vol1, so the busiest volume of the next most loaded controller (controller 2) is moved instead:

# /usr/lib/storpool/iscsi_balancer balance load

Strategy: load (move heaviest target from most-loaded to least-loaded controller)
Planned changes: 1 target(s)

Volume    Controller  Current Order  New Order
--------  ----------  -------------  ---------
web-vol2  2 -> 3      [2, 3, 1]      [3, 2, 1]

Run with --exec to apply. State will be auto-saved before execution.

Execute the balancing:

# /usr/lib/storpool/iscsi_balancer balance load --exec

Instead of executing, we can also print the equivalent storpool_q command (available for all strategies):

# /usr/lib/storpool/iscsi_balancer balance load --commands
storpool_q -d '{"commands":[{"targetSetControllers":{"volumeName":"web-vol2","controllers":[3,2,1]}}]}' iSCSIConfig

Manually moving a volume

To manually migrate a specific volume to a chosen controller, use balance move <volume> <controller>. The remaining controllers are appended to the list ordered from least to most loaded.

Print the plan (dry run):

# /usr/lib/storpool/iscsi_balancer balance move db-vol2 1

Strategy: manual (move a specific target to a chosen controller)
Planned changes: 1 target(s)

Volume   Controller  Current Order  New Order
-------  ----------  -------------  ---------
db-vol2  3 -> 1      [3, 2, 1]      [1, 3, 2]

Run with --exec to apply. State will be auto-saved before execution.

Execute the move:

# /usr/lib/storpool/iscsi_balancer balance move db-vol2 1 --exec

Saving and restoring state

Every time a balance command is executed with --exec, the tool first saves the current controller lists of all targets to a file named iscsi-state-<YYYYMMDD-HHMM>.json in the current working directory:

# /usr/lib/storpool/iscsi_balancer balance intrusive --exec
...
State saved to /root/iscsi-state-20251106-1700.json
Re-balance is successful!

The state can also be saved manually at any time:

# /usr/lib/storpool/iscsi_balancer state save
Current state saved to /root/iscsi-state-20251106-1700.json

If the result of a balancing operation is not satisfactory, the previous controller lists can be restored from the state file:

# /usr/lib/storpool/iscsi_balancer state restore /root/iscsi-state-20251106-1700.json
State has been restored from /root/iscsi-state-20251106-1700.json

Automatic session verification

For the operations that cause session reconnects (balance intrusive, balance load, balance move, and state restore), the tool automatically:

  1. Dumps the currently active iSCSI sessions to a file named iscsi-sessions-<YYYYMMDD-HHMM>.json in the current working directory;

  2. Applies the configuration change;

  3. Polls the active sessions (up to 40 times, one second apart) until every session from the dump has reconnected.

On success the sessions file is removed. If some sessions do not reconnect within the timeout, the tool reports them, keeps the sessions file, and the configuration change stays applied. The missing sessions can then be re-checked manually:

# /usr/lib/storpool/check_iscsi_sessions --mode check <iscsi-sessions-file.json>

balance backup does not disconnect any sessions, so no verification is performed for it.