Ejected disk

When a disk is ejected by some of the server instances and does not get back automatically:

1. Check if the drive is actually available.

Check if the drive is missing with storpool_initdisk --list --json |jq '.disks[] | select(.id==DISKID)':

# storpool_initdisk --list --json |jq '.disks[] | select(.id==118)'
 {
   "id": 118,
   "device": "/dev/sdp1",
   "isStorPool": true,
   "cluster": "b7ug.b",
   "nvme": false,
   "sectors": 15628048384,
   "type": "storpool",
   "typeId": 4,
   "meta": {
     "version": 65545,
     "instance": 3,
     "ssd": false,
     "wbc": false,
     "noFua": true,
     "noFlush": false,
     "noTrim": false,
     "testOverride": null,
     "diskMagickValue": 1615840159737,
     "journalMagickValue": 0,
     "entriesCount": 30600000,
     "objectsCount": 7650000
   }
 }

If the drive is missing, proceed to “Removal & balancing out”.

2. Check if this is a repeated event

In storpool -j disk list | jq '.data[] | select(.id==DISKID) | .testResults' you can see if the drive was ejected before and if the last automated test done by the server had any problems:

[root@storpool1 ~]# storpool -j disk list | jq '.data[] | select(.id==118) | .testResults'
{
  "dataCorruption": false,
  "failed": false,
  "readBandwidthThreshold": false,
  "readBps": 126000000,
  "readError": false,
  "readLatencyThreshold": false,
  "readMaxLat": 18,
  "stall": false,
  "timesTested": 12,
  "writeBandwidthThreshold": false,
  "writeBps": 137000000,
  "writeError": false,
  "writeLatencyThreshold": false,
  "writeMaxLat": 33
}

In this case, the drive has been ejected/tested 12 times. If the number of times is above 100, go to “Removal & balancing out”.

3. Check if the drive’s behavior is bad for the cluster

Check if the drive has been knownly stalling requests with grep storpool_server.*stalled /var/log/messages. If this is a regular occurence, see to “Removal & balancing out”.

Note

If all drives on the controller have stalled at the same time, look if there’s an issue with the controller itself.

Check if the drive’s behavior (latency-wise) is getting progressively worse, especially compared to the other drives on the node, in Analytics, in “System disks”->”SP disk stat”. If the drive’s latencies have started to go above some seconds, and this is a new behavior, see to “Removal & balancing out”.

4. Return drive to the cluster

If none of the above has led to removal of the drive, return to the cluster:

# storpool_initdisk -r /dev/sdXN   # where X is the drive letter and N is the partition number

After that, wait for the drive to show back in the cluster (storpool disk list) and for the recovery tasks to complete. If the drive is unable to get back for some reason, see to see to “Removal & balancing out”, otherwise END.

5. Removal & balancing out

At this step, we do know that the drive in question is not a good fit for StorPool and has to be removed.

5.1. Collect information on the drive

For any drive we remove, we need to provide two types of information to the customer:

  • Location/drive information - Node, disk ID, model, serial, slot (if we know it), and if possible, to enable to location LED;

  • Error information - any information we have on the problem with the drive, which is acceptable to them and their vendor to get a replacement.

5.1.1. Location information

Some data for any drive in StorPool can be seen with the diskid-helper and related tools:

Disk attached to Dell 3108-based controller:

[root@hc1 ~]# /usr/lib/storpool/diskid-helper /dev/sdb
MODEL=ST4000NM0023
SERIAL=Z1Z5621F
METHOD=PERCCLI_CMD
MODULE=megaraid_sas
[root@hc1 ~]# /usr/lib/storpool/storcli-helper.pl -p perccli64 /dev/sdb
SMARTCMD='smartctl -a -d megaraid,34 /dev/sdb'
ID_SERIAL='Z1Z5621F'
ID_MODEL='ST4000NM0023'
ID_ENCLOSURE='32'
ID_SLOT='2'
ID_CTRL='0'
ID_VDID='1'
[root@hc1 ~]# smartctl -a -d megaraid,34 /dev/sdb
smartctl 6.5 2016-05-07 r4318 [x86_64-linux-3.10.0-862.11.6.el7.x86_64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

Warning: DEFAULT entry missing in drive database file(s)
=== START OF INFORMATION SECTION ===
Vendor:               SEAGATE
Product:              ST4000NM0023
Revision:             GS0F
Compliance:           SPC-4
User Capacity:        4,000,787,030,016 bytes [4.00 TB]
Logical block size:   512 bytes
LU is fully provisioned
Rotation Rate:        7200 rpm
Form Factor:          3.5 inches
Logical Unit id:      0x5000c5005923ca73
Serial number:        Z1Z5621F
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Thu Sep 30 13:42:04 2021 BST
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Disabled or Not Supported

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:     28 C
Drive Trip Temperature:        60 C

Manufactured in week 31 of year 2014
Specified cycle count over device lifetime:  10000
Accumulated start-stop cycles:  101
Specified load-unload count over device lifetime:  300000
Accumulated load-unload cycles:  1930
Elements in grown defect list: 0

Vendor (Seagate) cache information
  Blocks sent to initiator = 180993466
  Blocks received from initiator = 2497577725
  Blocks read from cache and sent to initiator = 3544629222
  Number of read and write commands whose size <= segment size = 1020357238
  Number of read and write commands whose size > segment size = 1206

Vendor (Seagate/Hitachi) factory information
  number of hours powered up = 43603.95
  number of minutes until next internal SMART test = 58

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:   1693677694      145         0  1693677839        145    1262332.018           0
write:         0        0         0         0          0     301934.463           0
verify: 17510550        0         0  17510550          0          0.000           0

Non-medium error count:     1328

SMART Self-test log
Num  Test              Status                 segment  LifeTime  LBA_first_err [SK ASC ASQ]
     Description                              number   (hours)
# 1  Background long   Completed                  64   42181                 - [-   -    -]
# 2  Background short  Completed                  64      13                 - [-   -    -]
# 3  Reserved(7)       Completed                  48      13                 - [-   -    -]
# 4  Background short  Completed                  64       9                 - [-   -    -]
# 5  Background short  Completed                  64       7                 - [-   -    -]
# 6  Reserved(7)       Completed                  48       7                 - [-   -    -]
# 7  Background short  Completed                  64       3                 - [-   -    -]

Long (extended) Self Test duration: 32700 seconds [545.0 minutes]

In the example above, you can see that:

  • The drive is connected to a megaraid controller, that’s controlled via perccli;

  • The drive’s model is ST4000NM0023, the serial Z1Z5621F

  • It’s connlected on controller 0, enclosure 32, slot 2;

  • There are no defects in the growth defects list, but 145 slow error corrections.

Below, there are two more examples with shortened output - a directly attached drive and one via a HP controller:

[root@sof1 ~]# /usr/lib/storpool/diskid-helper /dev/sda
MODEL=INTEL_SSDSC2KB038T7
SERIAL=PHYS750100YN3P8EGN
METHOD=ATA_ID_CMD
MODULE=ahci

root@magnetic1.sjc:~# /usr/lib/storpool/diskid-helper /dev/sdo
MODEL=LOGICAL_VOLUME
SERIAL=PDNLL0CRHAN28K
METHOD=SCSI_ID_CMD
MODULE=hpsa
root@magnetic1.sjc:~# /usr/lib/storpool/hpssacli-helper /dev/sdo
ID_SERIAL='ZC1742B3'
ID_MODEL='ATA     MB4000GVYZK'
SMARTCMD='smartctl -a -d sat+cciss,8 /dev/sg17'

5.1.2. Error information

This information is needed by the customer to be able to deal with warranties and disk replacements. Not all customers require such information, but if we see any error in SMART that could be useful, it’s worth sharing with the customer to make their lives easier.

As SMART is not unified between SATA, SAS and NVMes, there’s no simple guide to the values that can be used to say “This drive is broken”. The following fields can be used:

  • 5 Reallocated_Sector_Ct : > 100

  • 187 Reported_Uncorrect : > 100

  • 197 Current_Pending_Sector : > 10

  • 198 Offline_Uncorrectable : > 10

  • 199 UDMA_CRC_Error_Count : > 0

  • Elements in grown defect list: > 100

5.2. Forget and start balancing

Removal of the drive is done with the following steps:

5.2.1. Remove drive from placement groups and from list in the cluster

From a node that has access to the API:

storpool disk XXX forget

5.2.2. Make sure the drive is flagged as ejected:

Run on the node with the drive, if the drive is writable:

storpool_initdisk -F XXX /dev/sdXN

This step is needed so if the node or the storpool_server process restarts before the drive is replaced, the storpool_server service will not try to add the drive in the cluster.

5.2.3. Clean up RAID VDs, if applicable

For any drive that’s with WBC in the RAID controller and has a journal, remove the VDs:

storcli64 /c0/v1235 del
storcli64 /c0/v1234 del

5.2.4. Start locate function, if available

storcli64 /c0/e252/s4 start locate
sas3ircu 0 locate 2:23 on

5.2.5. Balance out the drive (restore redundancy)

The basic steps are: run the balancer and commit, as per 15.  Rebalancing StorPool. There are two extra things to look for:

  • If there’s not enough space for restoring the redundancy, you can add -f 95 to try with filling the drives more;

  • If even with -f 95 the redundancy can’t be restored.

If any of the above is true, notify the customer immediately, especially in the case when redudancy can’t be restored, and look into other options.

5.3. Notify customer on the event

Open a ticket to the customer to notify them about the drive, that it needs a replacement, and if there are any issues with restoring redundancy.

5.4. Monitor progress

Montor the progress of the rebalance and see if there are any issues for it to complete.

watch -n10 -d 'storpool relocator status; storpool task list groupBy disk'

5.5. Notify customer that the rebalance is complete and the new drive is expected.

5.6. After the drive is replaced, disable locator lights.

storcli64 /c0/e252/s4 stop locate
sas3ircu 0 locate 2:23 off

Note

In some clusters, you also need to update the firmware of the new drive before adding it in the cluster.

See adding disk to a cluster for the remainder of the procedure when the drive is replaced.