Management configuration

Tip

Please consult with StorPool support before changing the management configuration defaults.

You can manage the internal cluster configuration parameters on the command line using the storpool mgmtConfig command.

Listing current configuration

To list the presently configured parameters:

# storpool mgmtConfig list
relocator on, interval 5.000 s
relocator transaction: min objects 320, max objects 4294967295
relocator recovery: max tasks per disk 2, max objects per disk 2400
relocator recovery objects trigger 32
relocator min free 150 GB
relocator max objects per HDD tail 0
balancer auto off, interval 5.000 s
snapshot delete interval 1.000 s
disks soft-eject interval 5.000 s
snapshot delayed delete off
snapshot dematerialize interval 1.000 s
mc owner check interval 2.000 s
mc autoreconcile interval 2.000 s
reuse server implicit on disk down disabled
max local recovery requests 1
max remote recovery requests 2
maintenance state production
max disk latency nvme 1000.000 ms
max disk latency ssd 1000.000 ms
max disk latency hdd 1000.000 ms
max disk latency journal 50.000 ms
backup template name backup_template
aggScoreSpace sameAg  99
aggScoreSpace suppress for disk full below 1%
aggScoreSpace restore for disk full above 2%
writeIopsWeight 1
writeBwWeight 1
trimIopsWeight 1
trimBwWeight 1

Local and remote recovery

The parameters of storpool mgmtConfig related to local and remote recovery are as follows:

maxLocalRecoveryRequests

Set max local recovery requests.

maxRemoteRecoveryRequests

Set max remote recovery requests.

Using these parameters you can set the number of parallel requests to issue while performing local or remote recovery, respectively. The values of the parameters should be between 1 and 64.

To set the default local recovery requests for all disks:

StorPool> mgmtConfig maxLocalRecoveryRequests 1
OK
StorPool> mgmtConfig maxRemoteRecoveryRequests 2
OK

For details on how you can override the values per disk, and also how to clear the overrides so that the defaults take precedence, see Setting recovery overrides.

An example use case would be the need to speed up or slow down a rebalancing or a remote transfer, based on the operational requirements at the time. For example, to lower the impact on latency-sensitive user operations, or decrease the time required for getting a cluster back to full redundancy, or getting a remote transfer completed faster.

These parameters are introduced with the 19.3 revision 19.01.2592.cf99471bd release.

Relocator parameters

The parameters of storpool mgmtConfig related to the relocator (see Rebalancing the cluster) are as follows:

relocatorInterval

Set relocator interval (in milliseconds); default is 5000.

relocatorMinTrObjects

Set min number of objects per relocator transaction.

relocatorMaxTrObjects

Set max number of objects per relocator transaction.

relocatorMaxRecoveryTasksPerDisk

Set max number of tasks per disk due to relocation.

relocatorMaxRecoveryObjectsPerDisk

Set max number of objects in recovery per disk due to relocation.

relocatorRecoveryObjectsTrigger

Set max number of objects in recovery for a disk to be usable by the relocator.

relocatorGBFreeBeforeAdd

Set relocator min free GB per disk before adding objects.

relocatorMaxTrObjectsPerHddTail

Set max relocating objects per HDD tail. 0 is unset, 1+ is number of objects.

Here are some examples. To change the interval (from the default 5 to 20 seconds, value is in milliseconds) for the relocator to check if there is new work to be done:

# storpool mgmtConfig relocatorInterval 20000
OK

To set a different than the default number of objects per disk (3200) in recovery at a time (see also Getting information about disk objects):

# storpool mgmtConfig relocatorMaxRecoveryObjectsPerDisk 2000 # value in number of objects per disk
OK

To change the default maximum number of recovery tasks per disk (2 tasks):

# storpool mgmtConfig relocatorMaxRecoveryTasksPerDisk 4 # value is number of tasks per disk - will set 4 tasks
OK

To change the minimum (default 320) or the maximum (default 4294967295) number of objects per transaction for the relocator:

# storpool mgmtConfig relocatorMaxTrObjects 2147483647
OK
# storpool mgmtConfig relocatorMinTrObjects 640
OK

To change the maximum number of objects per transaction per HDD tail drives:

# storpool mgmtConfig relocatorMaxTrObjectsPerHddTail 2

To change the maximum number of objects in recovery for a disk to be usable by the relocator (default 32):

# storpool mgmtConfig relocatorRecoveryObjectsTrigger 64

To update the free space threshold in GB after which the relocator will not be adding new tasks:

# storpool mgmtConfig relocatorGBFreeBeforeAdd 75 # value is in GB

Snapshots

The parameters of storpool mgmtConfig related to snapshots are as follows:

snapshotDeleteInterval

Set snapshot delete check interval (in milliseconds); ‘0’ for ‘off’.

delayedSnapshotDelete

Enable or disable delayed snapshot delete’; value should be ‘on’ or ‘off’.

To disable the deferred snapshot deletion (default on):

# storpool mgmtConfig delayedSnapshotDelete off
OK

When enabled, all snapshots with configured time to be deleted will be cleared at the configured date and time. To change the default check for new snapshots for deleting:

# storpool mgmtConfig snapshotDeleteInterval 1000

Soft ejection of disks

The parameters of storpool mgmtConfig related to soft ejection of disks are as follows:

disksSoftEjectInterval

Set disks soft eject check interval (in milliseconds).

To change the default interval between periodic checks whether disks marked for ejection can actually be ejected (5 sec.):

# storpool mgmtConfig disksSoftEjectInterval 20000 # value in ms - 20 sec.
OK

Snapshot dematerialization

Snapshot dematerialization checks and removes all objects that do not refer to any data - that is, no change in this object from the last snapshot (or ever). This helps reducing the number of used objects per disk in clusters with a large number of snapshots and a small number of changed blocks between the snapshots in the chain.

The parameters of storpool mgmtConfig related to snapshot dematerialization are as follows:

snapshotDematerializeInterval

Set snapshot dematerialize interval (in milliseconds); “0” for “off”.

Here is an example how to enable snapshot dematerialization or change the interval:

# storpool mgmtConfig snapshotDematerializeInterval 30000

Multi-cluster parameters

The parameters of storpool mgmtConfig related to multi-cluster setups (see Multi-site and multi-cluster) are as follows:

mcOwnerCheckInterval

Set multi-cluster owner check interval (in milliseconds). ‘0’ for ‘off’.

mcAutoReconcileInterval

Set multi-cluster auto reconcile interval (in milliseconds). ‘0’ for ‘off’.

To set or change the default multi-cluster owner check interval:

# storpool mcOwnerCheckInterval 2000

To set or change the default multi-cluster auto-reconcile interval:

# storpool mcAutoReconcileInterval 2000

Reusing server on disk failure

The parameters of storpool mgmtConfig related to reusing a server on disk failure are as follows:

reuseServerImplicitOnDiskDown

Enable reuse server implicit on disk down. Value should be ‘enable’ or ‘disable’.

If there is a disk down, and a new volume could not be allocated, on enabling the reuseServerImplicitOnDiskDown option the system will retry the new volume allocation as if the reuseServer parameter was specified; reuseServer can be set for volumes, snapshots, and templates on the CLI. This is helpful for minimum installation requirements with 3 nodes when one of the nodes or a disk is down.

To enable reusing a server on disk failure:

# storpool mgmtConfig reuseServerImplicitOnDiskDown enable

The only downside is that the volume will have two of its replicas on drives in the same server. When the missing node comes back a rebalancing will be required, so that all replicas created on the same server are re-distributed back on all nodes. A new needbalance alert will be raised for these occasions.

This option is disabled by default. Its history is as follows:

Default template

The parameters of storpool mgmtConfig related to templates are as follows:

backupTemplateName

Set a backup template.

defaultTemplateName

Set the default template.

To change the default template upon receiving a snapshot from a remote cluster, through the storpool_bridge service (was the now deprecated SP_BRIDGE_TEMPLATE):

# storpool mgmtConfig backupTemplateName all-flash # the all-flash template should exist
OK

Cluster maintenance mode

A full cluster maintenance mode is available for occasions involving full cluster related maintenance activities. An example would be a scheduled restart of a network switch that will be reported as missing network for all nodes in a cluster.

This mode does not perform any checks, and is mainly for informational purposes in order to sync context between customers and StorPool support teams. Full cluster maintenance mode could be used in addition to the per-node maintenance state explained above when necessary.

The parameters of storpool mgmtConfig related to maintenance mode are as follows:

maintenanceState

Set cluster maintenance state. Value should be ‘production’, ‘maintenance’ or ‘shutdown’.

To change the full cluster maintenance state to maintenance:

# storpool mgmtConfig maintenanceState maintenance
OK

To switch back into production state:

# storpool mgmtConfig maintenanceState production
OK

In case you only need to do this for a single node you can use storpool maintenance, as described in Maintenance mode.

Latency thresholds

Note

For individual per disk latency thresholds check Performance details section.

The parameters of storpool mgmtConfig related to latency thresholds are as follows:

maxDiskLatencies

Set global disk latency threshold for different disk types. You should first specify one of the nvme, ssd, hdd, or journal types. Then you should specify the threshold value is in milliseconds.

To define a global latency threshold before ejecting a HDD disk:

# storpool mgmtConfig maxDiskLatencies hdd 1000

To define a global latency threshold before ejecting a SSD drive:

# storpool mgmtConfig maxDiskLatencies ssd 1000

To define a global latency threshold before ejecting a NVMe drive:

# storpool mgmtConfig maxDiskLatencies nvme 1000

To define a global latency limit before ejecting a journal device:

# storpool mgmtConfig maxDiskLatencies journal 50

Aggregate scores

The parameters of storpool mgmtConfig related to aggregate scores are as follows:

aggScoreSpace

Set global disk aggregation score calculation parameters. You should first set one of the following parameters:

  • sameAg: value at which to enforce sameAG aggregation

  • suppressEnd: disk usage in percent until which to flatten aggScoreSpace to 1

  • restore: disk usage in percent at which to restore original aggScoreSpace behavior

Then you should specify a positive integer score.

To configure a different defaults for the disk space aggregation algorithm, use:

# storpool mgmtConfig aggScoreSpace suppressEnd 1
# storpool mgmtConfig aggScoreSpace restore 2
# storpool mgmtConfig aggScoreSpace sameAg 99

Note

These settings will be gradually modified in all production installations to new defaults that will be a lot less aggressive when a large amount of data gets deleted. With the new defaults the impact on user operations will be a lot smaller compared to the previous defaults with a mostly linear relation to the amount of data written and then freed from a disk:

# storpool mgmtConfig aggScoreSpace restore 95
# storpool mgmtConfig aggScoreSpace suppressEnd 90
# storpool mgmtConfig aggScoreSpace sameAg 10

Added with 21.0 revision 21.0.841.983f5880c release.

Volume ops weights

Four parameters of storpool mgmtConfig control how write and trim operations are counted against volume IOPS and bandwidth limits:

writeIopsWeight

Set write IOPS weight.

writeBwWeight

Set write bandwidth weight.

trimIopsWeight

Set trim IOPS weight.

trimBwWeight

Set trim bandwidth weight.

These settings apply only to volumes that have IOPS or bandwidth limits set. The default for each setting is 1 (counted the same as a read). The allowed range is 0.001 to 1000; trim weights may also be set to 0 (or -) so trims are not accounted for in the ops budget.

For use cases and examples with fio on a volume, see Volume Ops Weights.

To set the write IOPS and bandwidth weights:

# storpool mgmtConfig writeIopsWeight 0.5
OK
# storpool mgmtConfig writeBwWeight 2
OK

To set the trim IOPS and bandwidth weights (unlimited):

# storpool mgmtConfig trimIopsWeight 0 # same as setting '-'
OK

These parameters were added to StorPool in release 21.0 revision 21.0.3305.61d07a5b1.