Disk
Disk list main info
You can use the disk sub-menu to query and manage the available disks in the cluster.
To display all available disks in all server instances in the cluster:
# storpool disk list
disk | server | size | used | est.free | % | free entries | on-disk size | allocated objects | errors | flags
1101 | 11.1 | 893 GiB | 2.6 GiB | 857 GiB | 0 % | 3719946 | 664 KiB | 41000 / 930000 | 0 / 0 |
1102 | 11.1 | 446 GiB | 2.6 GiB | 424 GiB | 1 % | 1919946 | 664 KiB | 41000 / 480000 | 0 / 0 |
1103 | 11.0 | 893 GiB | 2.6 GiB | 857 GiB | 0 % | 3719948 | 660 KiB | 41000 / 930000 | 0 / 0 |
1104 | 11.0 | 446 GiB | 2.6 GiB | 424 GiB | 1 % | 1919946 | 664 KiB | 41000 / 480000 | 0 / 0 |
1105 | 11.0 | 446 GiB | 2.6 GiB | 424 GiB | 1 % | 1919947 | 664 KiB | 41000 / 480000 | 0 / 0 |
1111 | 11.1 | 930 GiB | 2.6 GiB | 893 GiB | 0 % | 974950 | 716 KiB | 41000 / 975000 | 0 / 0 |
1112 | 11.1 | 930 GiB | 2.6 GiB | 893 GiB | 0 % | 974949 | 736 KiB | 41000 / 975000 | 0 / 0 |
1113 | 11.1 | 930 GiB | 2.6 GiB | 893 GiB | 0 % | 974943 | 760 KiB | 41000 / 975000 | 0 / 0 |
1114 | 11.1 | 930 GiB | 2.6 GiB | 893 GiB | 0 % | 974937 | 844 KiB | 41000 / 975000 | 0 / 0 |
[snip]
1425 | 14.1 | 931 GiB | 2.6 GiB | 894 GiB | 0 % | 974980 | 20 MiB | 40100 / 975000 | 0 / 0 |
1426 | 14.1 | 931 GiB | 2.6 GiB | 894 GiB | 0 % | 974979 | 20 MiB | 40100 / 975000 | 0 / 0 |
----------------------------------------------------------------------------------------------------------------------------------------
47 | 8.0 | 30 TiB | 149 GiB | 29 TiB | 0 % | 53308967 | 932 MiB | 1844600 / 32430000 | 0 / 0 |
To mark a device as temporarily unavailable:
# storpool disk 1111 eject
OK
Ejecting a disk from the cluster will stop the data replication for this disk, but will keep the metadata about the placement groups in which it participated and the volume objects from these groups it contained.
Note
The command above will refuse to eject the disk if this operation
would lead to volumes or snapshots going into the down state
(usually when the last up-to-date copy for some parts of a
volume/snapshot is on this disk).
This disk will be shown as missing in the storpool disk list output, for
example:
# storpool disk list
disk | server | size | used | est.free | % | free entries | on-disk size | allocated objects | errors | flags
[snip]
1422 | 14.1 | - | - | - | - % | - | - | - / - | - / - |
[snip]
Attention
This operation leads to degraded redundancy for all volumes and snapshots that have data on the ejected disk.
Such a disk will not return by itself back into the cluster, and would have to
be manually reinserted by removing its EJECTED flag with
storpool_initdisk -r /dev/$path.
Disk list additional information
To display additional information about the disks:
# storpool disk list info
disk | server | device | model | serial | description | flags |
1101 | 11.1 | 0000:04:00.0-p1 | SAMSUNG MZQLB960HAJR-00007 | S437NF0M500149 | | S |
1102 | 11.1 | /dev/sdj1 | Micron_M500DC_MTFDDAK480MBB | 14250C6368E5 | | S |
1103 | 11.0 | /dev/sdi1 | SAMSUNG_MZ7LH960HAJR-00005 | S45NNE0M229767 | | S |
1104 | 11.0 | /dev/sdd1 | Micron_M500DC_MTFDDAK480MBB | 14250C63689B | | S |
1105 | 11.0 | /dev/sdc1 | Micron_M500DC_MTFDDAK480MBB | 14250C6368EC | | S |
1111 | 11.1 | /dev/sdl1 | Hitachi_HUA722010CLA330 | JPW9K0N13243ZL | | W |
1112 | 11.1 | /dev/sda1 | Hitachi_HUA722010CLA330 | JPW9J0N13LJEEV | | W |
1113 | 11.1 | /dev/sdb1 | Hitachi_HUA722010CLA330 | JPW9J0N13N694V | | W |
1114 | 11.1 | /dev/sdm1 | Hitachi_HUA722010CLA330 | JPW9K0N132R7HL | | W |
[snip]
1425 | 14.1 | /dev/sdm1 | Hitachi_HDS721050CLA360 | JP1532FR1BY75C | | W |
1426 | 14.1 | /dev/sdh1 | Hitachi_HUA722010CLA330 | JPW9K0N13RS95L | | W, J |
To set additional information for some of the disks shown in the output of
storpool disk list info:
# storpool disk 1111 description HBA2_port7
OK
# storpool disk 1104 description FAILING_SMART
OK
Disk list server internal information
To display internal statistics about each disk:
# storpool disk list internal
--------------------------------------------------------------------------------------------------------------------------------------------------------
| disk | server | aggregate scores | wbc pages | scrub bw | scrub ETA | last scrub completed |
--------------------------------------------------------------------------------------------------------------------------------------------------------
| 1101 | 11.1 | 0 | 0 | 0 | - + - / - | - | - | 2022-09-08 18:23:07 |
| 1102 | 11.1 | 0 | 0 | 0 | - + - / - | - | - | 2022-09-08 18:23:07 |
| 1103 | 11.0 | 0 | 0 | 0 | - + - / - | - | - | 2022-09-08 18:23:08 |
| 1104 | 11.0 | 0 | 0 | 0 | - + - / - | - | - | 2022-09-08 18:23:09 |
| 1105 | 11.1 | 0 | 0 | 0 | - + - / - | - | - | 2022-09-08 18:23:10 |
| 1111 | 11.0 | 0 | 0 | 0 | 0 + 0 / 2560 | - | - | 2022-09-08 18:23:12 |
| 1112 | 11.1 | 0 | 0 | 0 | 0 + 0 / 2560 | - | - | 2022-09-08 18:23:15 |
| 1113 | 11.1 | 0 | 0 | 0 | 0 + 0 / 2560 | - | - | 2022-09-08 18:23:17 |
| 1114 | 11.1 | 0 | 0 | 0 | 0 + 0 / 2560 | - | - | 2022-09-08 18:23:13 |
[snip]
| 1425 | 14.1 | 0 | 0 | 0 | 0 + 0 / 2560 | - | - | 2022-09-08 18:23:15 |
| 1426 | 14.1 | 0 | 0 | 0 | 0 + 0 / 2560 | - | - | 2022-09-08 18:23:19 |
--------------------------------------------------------------------------------------------------------------------------------------------------------
The sections in this output are as follows:
aggregate scoresInternal values representing how much data is about to be defragmented on the particular drive. Usually between 0 and 1, on heavily loaded clusters the rightmost column might get into the hundreds or even thousands if some drives are severely loaded.
wbc pagesInternal statistics for each drive that have the write back cache or journaling in StorPool enabled.
scrub bwThe scrubbing speed in MB/s.
scrub ETAApproximate time/date when the scrubbing operation will complete for this drive.
last scrub completedThe last time/date when the drive was scrubbed.
Note
The default installation includes a cron job on the management nodes that starts a scrubbing job for one drive per node. You can increase the number of disks that are scrubbing in parallel per node (the example is for four drives) by running the following:
# . /usr/lib/storpool/storpool_confget.sh
# storpool_q -d '{"set":{"scrubbingDiskPerNode":"4"}}' KV/Set/conf
And you can see the number of drives that are scrubbing per node with:
# . /usr/lib/storpool/storpool_confget.sh
# storpool_q KV/Get/conf/scrubbingDiskPerNode | jq -re '.data.pairs.scrubbingDiskPerNode'
To configure a local or remote recovery override for a particular
disk that is different than the overrides configured with mgmtConfig:
# storpool disk 1111 maxRecoveryRequestsOverride local 2
OK
# storpool disk 1111 maxRecoveryRequestsOverride remote 4
OK
To remove a configured override:
# storpool disk 1111 maxRecoveryRequestsOverride remote clear
OK
This will remove the override, and the default configured for the whole cluster at Management configuration will take precedence.
Disk list performance information
To display performance related in-server statistics for each disk use:
# storpool disk list perf
| latencies | thresholds | times exceeded | flags
disk | disk (avg) | disk (max) | jrn (avg) | jrn (max) | disk | journal | disk | journal |
2301 | 0.299ms | 0.400ms | - | - | 0.000ms | - | 0 | - |
2302 | 0.304ms | 0.399ms | - | - | 0.000ms | - | 0 | - |
2303 | 0.316ms | 0.426ms | - | - | 0.000ms | - | 0 | - |
[snip]
2621 | 4.376ms | 4.376ms | 0.029ms | 0.029ms | 0.000ms | 0.000ms | 0 | 0 |
2622 | 4.333ms | 4.333ms | 0.025ms | 0.025ms | 0.000ms | 0.000ms | 0 | 0 |
Note
Global latency thresholds are configured through the mgmtConfig section.
To configure a single disk latency threshold override use:
# storpool disk 2301 latencyLimitOverride disk 500
OK
# storpool disk list perf
| latencies | thresholds | times exceeded | flags
disk | disk (avg) | disk (max) | jrn (avg) | jrn (max) | disk | journal | disk | journal |
2301 | 0.119ms | 0.650ms | - | - | 500.000ms | - | 0 | - | D
[snip]
The D flag means there is a disk latency override, visible in the thresholds
section.
Similarly, to configure a single disk journal latency threshold override:
# storpool disk 2621 latencyLimitOverride journal 100
OK
storpool disk list perf
| latencies | thresholds | times exceeded | flags
disk | disk (avg) | disk (max) | jrn (avg) | jrn (max) | disk | journal | disk | journal |
[snip]
2621 | 8.489ms | 13.704ms | 0.052ms | 0.669ms | 0.000ms | 100.000ms | 0 | 0 | J
The J flag means there is a disk journal latency override, visible in the
thresholds section.
To override a single disk to no longer oblige the global limit:
# storpool disk 2301 latencyLimitOverride disk unlimited
OK
This will show the disk threshold as unlimited:
# storpool disk list perf
| latencies | thresholds | times exceeded | flags
disk | disk (avg) | disk (max) | jrn (avg) | jrn (max) | disk | journal | disk | journal |
2301 | 0.166ms | 0.656ms | - | - | unlimited | - | 0 | - | D
To clear an override and leave the global limit:
# storpool disk 2601 latencyLimitOverride disk off
OK
# storpool disk 2621 latencyLimitOverride journal off
OK
If a disk was ejected due to an excessive latency the server will keep a log from the last 128 requests sent to the disk, to list them use:
# storpool disk 2601 ejectLog
log creation time | time of first event
2022-03-31 17:50:16 | 2022-03-31 17:31:08 +761,692usec
req# | start | end | latency | addr | size | op
1 | +0us | +424us | 257us | 0x0000000253199000 | 128 KiB | DISK_OP_READ
[snip]
126 | +1147653582us | +1147653679us | 97us | 0x0000000268EBE000 | 12 KiB | DISK_OP_WRITE_FUA
127 | +1147654920us | +1147655192us | 272us | 0x0000000012961000 | 128 KiB | DISK_OP_WRITE_FUA
total | maxTotal | limit | generation | times exceeded (for this eject)
23335us | 100523us | 1280us | 15338 | 1
The same data is available if a disk journal was ejected after breaching the threshold with:
# storpool disk 2621 ejectLog journal
[snip]
(the output is similar to that of the disk ejectLog above)
Ejecting disks and internal server tests
When a server controlling a disk notices some issues with it (like a write error, or stalled request above a predefined threshold), the disk is also marked as “test pending”. This happens when there are many transient errors when a disk drive (or its controller) stalls a request for more than a predefined threshold.
Note
See also Automatic drive tests.
An eject option is available for manually initiating such a test, which will flag the disk that it requires test and will eject it. The server instance will then perform a quick set of non-intrusive read-write tests on this disk and will return it back into the cluster if all tests did well; for example:
# storpool disk 2331 eject test
OK
The tests are done usually couple of seconds up to a minute, to check the results from the last test:
# storpool disk 2331 testInfo
times tested | test pending | read speed | write speed | read max latency | write max latency | failed
1 | no | 1.0 GiB/sec | 971 MiB/sec | 8 msec | 4 msec | no
If the disk was already marked for testing the option “now” will skip the test on the next attempt to re-open the disk:
# storpool disk 2301 eject now
OK
Attention
Note that this is exactly the same as “eject”, the disk would have to be manually returned into the cluster.
To mark a disk as unavailable by first re-balancing all data out to the other disks in the cluster and only then eject it:
# storpool disk 1422 softEject
OK
Balancer auto mode currently OFF. Must be ON for soft-eject to complete.
Note
This option requires StorPool balancer to be started after the above was issued; for details, see Balancer.
To remove a disk from the list of reported disks and all placement groups it participates in:
# storpool disk 1422 forget
OK
To get detailed information about given disk:
# storpool disk 1101 info
agAllocated | agCount | agFree | agFreeing | agFull | agMaxSizeFull | agMaxSizePartial | agPartial
7 | 462 | 455 | 1 | 0 | 0 | 1 | 1
entriesAllocated | entriesCount | entriesFree | sectorsCount
50 | 1080000 | 1079950 | 501215232
objectsAllocated | objectsCount | objectsFree | objectStates
18 | 270000 | 269982 | ok:18
serverId | 1
id | objectsCount | onDiskSize | storedSize | objectStates
#bad_id | 1 | 0 B | 0 B | ok:1
#clusters | 1 | 8.0 KiB | 768 B | ok:1
#drive_state | 1 | 8.0 KiB | 4.0 B | ok:1
#drives | 1 | 100 KiB | 96 KiB | ok:1
#iscsi_config | 1 | 12 KiB | 8.0 KiB | ok:1
[snip]
To get detailed information about the objects on a particular disk:
# storpool disk 1101 list
object name | stored size | on-disk size | data version | object state | parent volume
#bad_id:0 | 0 B | 0 B | 1480:2485 | ok (1) |
#clusters:0 | 768 B | 8.0 KiB | 711:992 | ok (1) |
#drive_state:0 | 4.0 B | 8.0 KiB | 1475:2478 | ok (1) |
#drives:0 | 96 KiB | 100 KiB | 1480:2484 | ok (1) |
[snip]
test:4094 | 0 B | 0 B | 0:0 | ok (1) |
test:4095 | 0 B | 0 B | 0:0 | ok (1) |
----------------------------------------------------------------------------------------------------
4115 objects | 394 KiB | 636 KiB | | |
To get detailed information about the active requests that the disk is performing at the moment:
# storpool disk 1101 activeRequests
-----------------------------------------------------------------------------------------------------------------------------------
| request ID | request IDX | volume | address | size | op | time active |
-----------------------------------------------------------------------------------------------------------------------------------
| 9226469746279625682:285697101441249070 | 9 | testvolume | 85276782592 | 4.0 KiB | read | 0 msec |
| 9226469746279625682:282600876697431861 | 13 | testvolume | 96372936704 | 4.0 KiB | read | 0 msec |
| 9226469746279625682:278097277070061367 | 19 | testvolume | 46629707776 | 4.0 KiB | read | 0 msec |
| 9226469746279625682:278660227023482671 | 265 | testvolume | 56680042496 | 4.0 KiB | write | 0 msec |
-----------------------------------------------------------------------------------------------------------------------------------
To issue retrim operation on a disk (available for SSD disks only):
# storpool disk 1101 retrim
OK
To start, pause, or continue a scrubbing operation for a disk:
# storpool disk 1101 scrubbing start
OK
# storpool disk 1101 scrubbing pause
OK
# storpool disk 1101 scrubbing continue
OK
Note
Use storpool disk list internal to check the status of a running
scrub operation or when was the last completed scrubbing operation for
this disk.