Disk
Command parameters
You can perform disk operations on the command line using the storpool disk command.
Common parameters
When there is no disk ID specified to the storpool disk command you can use only the list parameter.
This parameter has additional arguments - info, internal, and perf.
For more information about the list parameter and its arguments, see Listings disks.
Disk-specific parameters
When there is a disk ID specified to the storpool disk command you can use the following parameters:
activeRequestsShow currently active Storpool requests for this disk; see Active requests.
descriptionAdd a description for a disk; see Setting disk description.
ejectEject a disk; see Ejecting disks and internal server tests.
ejectLogList disk eject log; see Setting latencies.
forgetForget a disk; see Forgetting a disk.
ignoreTry to boot cluster by ignoring this disk. Warning: possible data corruption.
latencyLimitsOverrideOverride options for this disk’s (and its journal’s) max acceptable IO latencies; see Setting latencies.
listList objects (logical elements of which a volume is comprised) on the specified disk.
maxRecoveryRequestsOverrideOverride options for this disk’s max local and remote recovery requests; see Setting recovery overrides.
perfInfoGet performance information about the specified disk.
retrimIssue a retrim operation on a disk; see Retrim operation.
scrubbingManage scrubbing; see Scrubbing operations.
softEjectPerform soft eject for the specified disk; see Soft ejecting.
testInfoDisk test information; see Internal server tests.
Listings disks
Main information
You can use the disk sub-menu to query and manage the available disks in the cluster.
To display all available disks in all server instances in the cluster:
# storpool disk list
disk | server | size | used | est.free | % | free entries | on-disk size | allocated objects | errors | flags
1101 | 11.1 | 893 GiB | 2.6 GiB | 857 GiB | 0 % | 3719946 | 664 KiB | 41000 / 930000 | 0 / 0 |
1102 | 11.1 | 446 GiB | 2.6 GiB | 424 GiB | 1 % | 1919946 | 664 KiB | 41000 / 480000 | 0 / 0 |
1103 | 11.0 | 893 GiB | 2.6 GiB | 857 GiB | 0 % | 3719948 | 660 KiB | 41000 / 930000 | 0 / 0 |
1104 | 11.0 | 446 GiB | 2.6 GiB | 424 GiB | 1 % | 1919946 | 664 KiB | 41000 / 480000 | 0 / 0 |
1105 | 11.0 | 446 GiB | 2.6 GiB | 424 GiB | 1 % | 1919947 | 664 KiB | 41000 / 480000 | 0 / 0 |
1111 | 11.1 | 930 GiB | 2.6 GiB | 893 GiB | 0 % | 974950 | 716 KiB | 41000 / 975000 | 0 / 0 |
1112 | 11.1 | 930 GiB | 2.6 GiB | 893 GiB | 0 % | 974949 | 736 KiB | 41000 / 975000 | 0 / 0 |
1113 | 11.1 | 930 GiB | 2.6 GiB | 893 GiB | 0 % | 974943 | 760 KiB | 41000 / 975000 | 0 / 0 |
1114 | 11.1 | 930 GiB | 2.6 GiB | 893 GiB | 0 % | 974937 | 844 KiB | 41000 / 975000 | 0 / 0 |
[snip]
1425 | 14.1 | 931 GiB | 2.6 GiB | 894 GiB | 0 % | 974980 | 20 MiB | 40100 / 975000 | 0 / 0 |
1426 | 14.1 | 931 GiB | 2.6 GiB | 894 GiB | 0 % | 974979 | 20 MiB | 40100 / 975000 | 0 / 0 |
----------------------------------------------------------------------------------------------------------------------------------------
47 | 8.0 | 30 TiB | 149 GiB | 29 TiB | 0 % | 53308967 | 932 MiB | 1844600 / 32430000 | 0 / 0 |
If a disk is ejected (see Ejecting disks and internal server tests)
it will be shown as missing in the storpool disk list output, for
example:
# storpool disk list
disk | server | size | used | est.free | % | free entries | on-disk size | allocated objects | errors | flags
[snip]
1422 | 14.1 | - | - | - | - % | - | - | - / - | - / - |
[snip]
Additional details
To display additional information about the disks:
# storpool disk list info
disk | server | device | model | serial | description | flags |
1101 | 11.1 | 0000:04:00.0-p1 | SAMSUNG MZQLB960HAJR-00007 | S437NF0M500149 | | S |
1102 | 11.1 | /dev/sdj1 | Micron_M500DC_MTFDDAK480MBB | 14250C6368E5 | | S |
1103 | 11.0 | /dev/sdi1 | SAMSUNG_MZ7LH960HAJR-00005 | S45NNE0M229767 | | S |
1104 | 11.0 | /dev/sdd1 | Micron_M500DC_MTFDDAK480MBB | 14250C63689B | | S |
1105 | 11.0 | /dev/sdc1 | Micron_M500DC_MTFDDAK480MBB | 14250C6368EC | | S |
1111 | 11.1 | /dev/sdl1 | Hitachi_HUA722010CLA330 | JPW9K0N13243ZL | | W |
1112 | 11.1 | /dev/sda1 | Hitachi_HUA722010CLA330 | JPW9J0N13LJEEV | | W |
1113 | 11.1 | /dev/sdb1 | Hitachi_HUA722010CLA330 | JPW9J0N13N694V | | W |
1114 | 11.1 | /dev/sdm1 | Hitachi_HUA722010CLA330 | JPW9K0N132R7HL | | W |
[snip]
1425 | 14.1 | /dev/sdm1 | Hitachi_HDS721050CLA360 | JP1532FR1BY75C | | W |
1426 | 14.1 | /dev/sdh1 | Hitachi_HUA722010CLA330 | JPW9K0N13RS95L | | W, J |
You can set what information appears in the description field as described in Setting disk description.
Internal details
To display internal statistics about each disk:
# storpool disk list internal
--------------------------------------------------------------------------------------------------------------------------------------------------------
| disk | server | aggregate scores | wbc pages | scrub bw | scrub ETA | last scrub completed |
--------------------------------------------------------------------------------------------------------------------------------------------------------
| 1101 | 11.1 | 0 | 0 | 0 | - + - / - | - | - | 2022-09-08 18:23:07 |
| 1102 | 11.1 | 0 | 0 | 0 | - + - / - | - | - | 2022-09-08 18:23:07 |
| 1103 | 11.0 | 0 | 0 | 0 | - + - / - | - | - | 2022-09-08 18:23:08 |
| 1104 | 11.0 | 0 | 0 | 0 | - + - / - | - | - | 2022-09-08 18:23:09 |
| 1105 | 11.1 | 0 | 0 | 0 | - + - / - | - | - | 2022-09-08 18:23:10 |
| 1111 | 11.0 | 0 | 0 | 0 | 0 + 0 / 2560 | - | - | 2022-09-08 18:23:12 |
| 1112 | 11.1 | 0 | 0 | 0 | 0 + 0 / 2560 | - | - | 2022-09-08 18:23:15 |
| 1113 | 11.1 | 0 | 0 | 0 | 0 + 0 / 2560 | - | - | 2022-09-08 18:23:17 |
| 1114 | 11.1 | 0 | 0 | 0 | 0 + 0 / 2560 | - | - | 2022-09-08 18:23:13 |
[snip]
| 1425 | 14.1 | 0 | 0 | 0 | 0 + 0 / 2560 | - | - | 2022-09-08 18:23:15 |
| 1426 | 14.1 | 0 | 0 | 0 | 0 + 0 / 2560 | - | - | 2022-09-08 18:23:19 |
--------------------------------------------------------------------------------------------------------------------------------------------------------
The sections in this output are as follows:
Column |
Description |
|---|---|
|
Internal values representing how much data is about to be defragmented on the particular drive. Usually between 0 and 1, on heavily loaded clusters the rightmost column might get into the hundreds or even thousands if some drives are severely loaded. |
|
Internal statistics for each drive that have the write back cache or journaling in StorPool enabled. |
|
The scrubbing speed in MB/s. |
|
Approximate time/date when the scrubbing operation will complete for this drive. |
|
The last time/date when the drive was scrubbed. |
Note that the default installation includes a cron job on the management nodes that starts a scrubbing job for one drive per node. You can increase the number of disks that are scrubbing in parallel per node (the example is for four drives) by running the following:
# . /usr/lib/storpool/storpool_confget.sh
# storpool_q -d '{"set":{"scrubbingDiskPerNode":"4"}}' KV/Set/conf
And you can see the number of drives that are scrubbing per node with:
# . /usr/lib/storpool/storpool_confget.sh
# storpool_q KV/Get/conf/scrubbingDiskPerNode | jq -re '.data.pairs.scrubbingDiskPerNode'
To configure a local or remote recovery override, see Setting recovery overrides.
Performance details
To display performance related in-server statistics for each disk use:
# storpool disk list perf
| latencies | thresholds | times exceeded | flags
disk | disk (avg) | disk (max) | jrn (avg) | jrn (max) | disk | journal | disk | journal |
2301 | 0.299ms | 0.400ms | - | - | 0.000ms | - | 0 | - |
2302 | 0.304ms | 0.399ms | - | - | 0.000ms | - | 0 | - |
2303 | 0.316ms | 0.426ms | - | - | 0.000ms | - | 0 | - |
[snip]
2621 | 4.376ms | 4.376ms | 0.029ms | 0.029ms | 0.000ms | 0.000ms | 0 | 0 |
2622 | 4.333ms | 4.333ms | 0.025ms | 0.025ms | 0.000ms | 0.000ms | 0 | 0 |
More information:
For details on configuring global latency thresholds using storpool mgmtConfig, see Latency thresholds.
To configure latencies with the latencyLimitsOverride argument and see how the results appear in the disk performance details, see Setting latencies.
Ejecting disks and internal server tests
Ejecting disks
To mark a device as temporarily unavailable:
# storpool disk 1111 eject
OK
Ejecting a disk from the cluster will stop the data replication for this disk, but will keep the metadata about the placement groups in which it participated and the volume objects from these groups it contained (see Placement groups and Volumes).
Note
The command above will refuse to eject the disk if this operation
would lead to volumes or snapshots going into the down state
(usually when the last up-to-date copy for some parts of a
volume/snapshot is on this disk).
This disk will be shown as missing in the storpool disk list output, see Listings disks.
Attention
This operation leads to degraded redundancy for all volumes and snapshots that have data on the ejected disk.
Such a disk will not return by itself back into the cluster, and would have to
be manually reinserted by removing its EJECTED flag with
storpool_initdisk -r /dev/$path (see Device preparation options (storpool_initdisk)).
Soft ejecting
To mark a disk as unavailable by first rebalancing all data out to the other disks in the cluster and only then eject it:
# storpool disk 1422 softEject
OK
Balancer auto mode currently OFF. Must be ON for soft-eject to complete.
Note
This option requires StorPool balancer to be started after the above was issued; for details, see Balancer.
Internal server tests
When a server controlling a disk notices some issues with it (like a write error, or stalled request above a predefined threshold), the disk is also marked as “test pending”. This happens when there are many transient errors when a disk drive (or its controller) stalls a request for more than a predefined threshold.
Note
See also Automatic drive tests.
An eject option is available for manually initiating such a test, which will flag the disk that it requires test and will eject it. The server instance will then perform a quick set of non-intrusive read-write tests on this disk and will return it back into the cluster if all tests did well; for example:
# storpool disk 2331 eject test
OK
The tests are done usually couple of seconds up to a minute, to check the results from the last test:
# storpool disk 2331 testInfo
times tested | test pending | read speed | write speed | read max latency | write max latency | failed
1 | no | 1.0 GiB/sec | 971 MiB/sec | 8 msec | 4 msec | no
If the disk was already marked for testing the option “now” will skip the test on the next attempt to re-open the disk:
# storpool disk 2301 eject now
OK
Attention
Note that this is exactly the same as “eject”, the disk would have to be manually returned into the cluster.
Other disk operations
Forgetting a disk
To remove a disk from the list of reported disks and all placement groups it participates in:
# storpool disk 1422 forget
OK
Getting detailed disk information
To get detailed information about given disk:
# storpool disk 1101 info
agAllocated | agCount | agFree | agFreeing | agFull | agMaxSizeFull | agMaxSizePartial | agPartial
7 | 462 | 455 | 1 | 0 | 0 | 1 | 1
entriesAllocated | entriesCount | entriesFree | sectorsCount
50 | 1080000 | 1079950 | 501215232
objectsAllocated | objectsCount | objectsFree | objectStates
18 | 270000 | 269982 | ok:18
serverId | 1
id | objectsCount | onDiskSize | storedSize | objectStates
#bad_id | 1 | 0 B | 0 B | ok:1
#clusters | 1 | 8.0 KiB | 768 B | ok:1
#drive_state | 1 | 8.0 KiB | 4.0 B | ok:1
#drives | 1 | 100 KiB | 96 KiB | ok:1
#iscsi_config | 1 | 12 KiB | 8.0 KiB | ok:1
[snip]
Getting information about disk objects
To get detailed information about the objects (fixed-size small chunks of data into which a StorPool volume is split) on a particular disk:
# storpool disk 1101 list
object name | stored size | on-disk size | data version | object state | parent volume
#bad_id:0 | 0 B | 0 B | 1480:2485 | ok (1) |
#clusters:0 | 768 B | 8.0 KiB | 711:992 | ok (1) |
#drive_state:0 | 4.0 B | 8.0 KiB | 1475:2478 | ok (1) |
#drives:0 | 96 KiB | 100 KiB | 1480:2484 | ok (1) |
[snip]
test:4094 | 0 B | 0 B | 0:0 | ok (1) |
test:4095 | 0 B | 0 B | 0:0 | ok (1) |
----------------------------------------------------------------------------------------------------
4115 objects | 394 KiB | 636 KiB | | |
Active requests
To get detailed information about the active requests that a disk is performing at the moment:
# storpool disk 1101 activeRequests
--------------------------------------------------------------------------------------------------------------------------------------------
| request ID | request IDX | volume | address | size | op | time active |
--------------------------------------------------------------------------------------------------------------------------------------------
| 9226469746279625682:285697101441249070 | 9 | testvolume | 85276782592 | 4.0 KiB | read | 0 msec |
| 9226469746279625682:282600876697431861 | 13 | testvolume | 96372936704 | 4.0 KiB | read | 0 msec |
| 9226469746279625682:278097277070061367 | 19 | testvolume | 46629707776 | 4.0 KiB | read | 0 msec |
| 9226469746279625682:278660227023482671 | 265 | testvolume | 56680042496 | 4.0 KiB | write | 0 msec |
--------------------------------------------------------------------------------------------------------------------------------------------
Retrim operation
To issue retrim operation on a disk (available for SSD disks only):
# storpool disk 1101 retrim
OK
Scrubbing operations
To start, pause, or continue a scrubbing operation for a specific disk:
# storpool disk 1101 scrubbing start
OK
# storpool disk 1101 scrubbing pause
OK
# storpool disk 1101 scrubbing continue
OK
Note
Use storpool disk list internal (see Internal details) to check the status of a running
scrub operation or when was the last completed scrubbing operation for
this disk.
Setting disk description
To set description information for a disk (shown in the output of
storpool disk list info, see Additional details)
you should specify the disk ID, the description argument, and the corresponding
text:
# storpool disk 1111 description HBA2_port7
OK
# storpool disk 1104 description FAILING_SMART
OK
Setting recovery overrides
To configure a local or remote recovery override for a particular
disk that is different than the overrides configured with mgmtConfig:
# storpool disk 1111 maxRecoveryRequestsOverride local 2
OK
# storpool disk 1111 maxRecoveryRequestsOverride remote 4
OK
To remove a configured override:
# storpool disk 1111 maxRecoveryRequestsOverride remote clear
OK
This will remove the override, and the default configured for the whole cluster with the storpool mgmtConfig command (see Management configuration) will take precedence.
Setting latencies
To configure a single disk latency threshold override use:
# storpool disk 2301 latencyLimitsOverride disk 500
OK
# storpool disk list perf
| latencies | thresholds | times exceeded | flags
disk | disk (avg) | disk (max) | jrn (avg) | jrn (max) | disk | journal | disk | journal |
2301 | 0.119ms | 0.650ms | - | - | 500.000ms | - | 0 | - | D
[snip]
The D flag means there is a disk latency override, visible in the thresholds
section.
Similarly, to configure a single disk journal latency threshold override:
# storpool disk 2621 latencyLimitsOverride journal 100
OK
storpool disk list perf
| latencies | thresholds | times exceeded | flags
disk | disk (avg) | disk (max) | jrn (avg) | jrn (max) | disk | journal | disk | journal |
[snip]
2621 | 8.489ms | 13.704ms | 0.052ms | 0.669ms | 0.000ms | 100.000ms | 0 | 0 | J
The J flag means there is a disk journal latency override, visible in the
thresholds section.
To override a single disk to no longer oblige the global limit:
# storpool disk 2301 latencyLimitsOverride disk unlimited
OK
This will show the disk threshold as unlimited:
# storpool disk list perf
| latencies | thresholds | times exceeded | flags
disk | disk (avg) | disk (max) | jrn (avg) | jrn (max) | disk | journal | disk | journal |
2301 | 0.166ms | 0.656ms | - | - | unlimited | - | 0 | - | D
To clear an override and leave the global limit:
# storpool disk 2601 latencyLimitsOverride disk off
OK
# storpool disk 2621 latencyLimitsOverride journal off
OK
If a disk was ejected due to an excessive latency the server will keep a log from the last 128 requests sent to the disk, to list them use:
# storpool disk 2601 ejectLog
log creation time | time of first event
2022-03-31 17:50:16 | 2022-03-31 17:31:08 +761,692usec
req# | start | end | latency | addr | size | op
1 | +0us | +424us | 257us | 0x0000000253199000 | 128 KiB | DISK_OP_READ
[snip]
126 | +1147653582us | +1147653679us | 97us | 0x0000000268EBE000 | 12 KiB | DISK_OP_WRITE_FUA
127 | +1147654920us | +1147655192us | 272us | 0x0000000012961000 | 128 KiB | DISK_OP_WRITE_FUA
total | maxTotal | limit | generation | times exceeded (for this eject)
23335us | 100523us | 1280us | 15338 | 1
The same data is available if a disk journal was ejected after breaching the threshold with:
# storpool disk 2621 ejectLog journal
[snip]
(the output is similar to that of the disk ejectLog above)
More information
You can also view and manage disks using the Graphical user interface and the REST API.