Cleanup of stale backup volumes and snapshots

Introduction

During VM backups, OnApp temporarily creates backup volumes and snapshots, which are usually automatically deleted by the controller after the backup is completed. Under specific conditions, however, the deletion procedure fails and the temporary volumes and snapshots created during the backup are left unused and only taking up free space. Most cases are already covered through workarounds in LVMSP, but a small portion still requires manual intervention.

Volumes attached to nodes

Start with detaching and deleting all backup volumes and snapshots that are more than 2 days old, by executing the following code snippet on a node with installed storpool_mgmt:

while read -u 4 s; do
  v="${s%-SNP}"
  set -x
  storpool detach volume "$v" all && \
  storpool volume "$v" delete "$v" && \
  storpool snapshot "$s" delete "$s"
  set +x;
done 4< <(storpool_req SnapshotsList|jq -r "map(select(.name|contains(\":backup-\"))|select(.creationTimestamp<$(date --date='2 days ago' +%s)))[]|.name")

If the above code snippet fails for different volumes, follow through this page depending on the kind of node the volumes are attached on.

Note

The above code snippet takes older than 2 days snapshots with names ending with ‘-SNP’ from storpool -j snapshot list (-j is for JSON output), detaches and removes volumes without it, and finally deletes the parent snapshot ending with ‘-SNP’. The commands issued in this sequence are the regular storpool detach volume <volume_name> all, storpool volume <volume_name> delete <volume_name> and storpool snapshot <snapshot_name> delete <snapshot_name>.

Volumes exported through iSCSI (Hypervisor nodes)

Look for the Target ID (TID) of the iSCSI export:

tgtadm --mode target --op show | grep 'Target.*backup-xxxxxxxxxxxxxx'

By checking the iSCSI exports for the backup volumes in question, note the TID (Target $TID: iqn…) for the corresponding entry, for example:

# tgtadm --mode target --op show | grep 'Target.*backup-fn7giuli19bswq'
...
Target 9: iqn.2019-03-07:onapp.com:backup-fn7giuli19bswq
...

On newer versions of OnApp, it is possible that there will be no output like in the above example. In that case, try running tgtadm without grep:

tgtadm --mode target --op show

Example output (note the second “Backing store path” line):

...
Target 9: iqn.2019-03-07:onapp.com:snapshot-ohdriiwuvbnssx
  System information:
      Driver: iscsi
      State: ready
  I_T nexus information:
  LUN information:
      LUN: 0
          Type: controller
          SCSI ID: IET     00010000
          SCSI SN: beaf10
          Size: 0 MB, Block size: 1
          Online: Yes
          Removable media: No
          Prevent removal: No
          Readonly: No
          Backing store type: null
          Backing store path: None
          Backing store flags:
      LUN: 1
          Type: disk
          SCSI ID: IET     00010001
          SCSI SN: beaf11
          Size: 429497 MB, Block size: 512
          Online: Yes
          Removable media: No
          Prevent removal: No
          Readonly: No
          Backing store type: rdwr
          Backing store path: /dev/onapp-qkwnlpjsuvxhox/backup-fn7giuli19bswq
          Backing store flags:
  Account information:
  ACL information:
      10.200.1.40
...

If the volume appears in the target, then it needs to be removed.

Proceed with the removal of the iSCSI target:

tgtadm --mode target --op update --tid=$TID -n state -v offline
tgtadm --mode target --op delete --tid=$TID

You can re-run the code snippet that does the cleanup at this point or do it after going through all the backup volumes in question.

Volumes held by the device mappper (Backup nodes)

Check for mounted volumes and un-mount:

mount | grep backup-xxxxxxxxxxxxxx | awk '{print $3}' | xargs umount -v

List the device mapper entries which are using the volumes and preventing their detachment:

for s in /dev/storpool/*; do echo $s; sd=`readlink $s`; for dm in `ls /sys/block/${sd##*/}/holders`; do echo $dm; done ;done

The volumes in question should have entries with device IDs (dm-$DID) below them, for example:

/dev/storpool/onapp-5ef2xsbvljxc2f:backup-bqaauthhrsndqk
dm-5
/dev/storpool/onapp-5ef2xsbvljxc2f:backup-cjttqdnpoopinm
dm-6
/dev/storpool/onapp-5ef2xsbvljxc2f:backup-dymkzwrgmunegq
dm-8
/dev/storpool/onapp-5ef2xsbvljxc2f:backup-zbpzhztdfidpxp
dm-3
/dev/storpool/onapp-5ef2xsbvljxc2f:backup-okcdxzqokcxidc
dm-2
/dev/storpool/onapp-5ef2xsbvljxc2f:backup-wyfrhcqwmnrqfm
dm-4

If there are no running processes, the device mapper can be cleaned with the following command:

dmsetup remove /dev/dm-$DID

After removing the device mapper entries with dmsetup, there will be two symlinks left for each entry, one linking a device file to the backup volume and one with a similar device name linking to the raw StorPool disk device (/dev/sp-..). Both can be listed using:

find /dev/sp* -type l -exec ls -l {} +

Example output:

lrwxrwxrwx 1 root root      9 Nov 30 10:18 /dev/sp0p -> /dev/sp-0
lrwxrwxrwx 1 root root     56 Nov 30 10:18 /dev/sp0p1 -> /dev/mapper/onapp-5ef2xsbvljxc2f:backup-okcdxzqokcxidcX1
lrwxrwxrwx 1 root root      9 Nov 15 13:39 /dev/sp1p -> /dev/sp-1
lrwxrwxrwx 1 root root     56 Nov 15 13:39 /dev/sp1p1 -> /dev/mapper/onapp-5ef2xsbvljxc2f:backup-zbpzhztdfidpxpX1
lrwxrwxrwx 1 root root      9 Nov 30 12:04 /dev/sp2p -> /dev/sp-2
lrwxrwxrwx 1 root root     56 Nov 30 12:04 /dev/sp2p1 -> /dev/mapper/onapp-5ef2xsbvljxc2f:backup-wyfrhcqwmnrqfmX1
lrwxrwxrwx 1 root root      9 Dec 10 12:35 /dev/sp3p -> /dev/sp-3
lrwxrwxrwx 1 root root     56 Dec 10 12:35 /dev/sp3p1 -> /dev/mapper/onapp-5ef2xsbvljxc2f:backup-bqaauthhrsndqkX1
lrwxrwxrwx 1 root root      9 Dec 10 13:17 /dev/sp4p -> /dev/sp-4
lrwxrwxrwx 1 root root     56 Dec 10 13:17 /dev/sp4p1 -> /dev/mapper/onapp-5ef2xsbvljxc2f:backup-cjttqdnpoopinmX1
lrwxrwxrwx 1 root root      9 Nov 30 12:17 /dev/sp6p -> /dev/sp-6
lrwxrwxrwx 1 root root     56 Nov 30 12:17 /dev/sp6p1 -> /dev/mapper/onapp-5ef2xsbvljxc2f:backup-dymkzwrgmunegqX1

The pair of symlinks in question can be safely removed via rm.

When there are no holders left for the backup volumes in question, proceed with re-running the code snippet that does the cleanup.