VolumeCare

1. Overview

VolumeCare is a StorPool service that creates and manages atomic consistent snapshots of volumes, based on defined retention policies. Its main objective is to provide the ability to roll back to a previous state of a volume.

The service also detects if multiple volumes belong to the same virtual machine (based on tags added by the orchestration/integration) and creates crash-consistent snapshots for the whole VM. It also allows the administrator to revert based on a virtual machine ID instead of volume by volume.

VolumeCare supports working modes and policies for multiple cluster configurations. With theese a backup cluster can be used to store the snapshots instead or along with the primary cluster. In this configuration there will be a volumecare service running in each of the clusters.

2. General configuration

The VolumeCare configuration file consists of 4 types of sections: format, volumecare, policy and template sections.

The [format] section is mandatory and describes the version of the format for the configuration through the version=X.Y option. Currently, only version 1.0 is available.

The [volumecare] section describes working parameters for the service.

A mode option should be specified which defines the working routine of the service and can be one of the following:

  • normal - for a single cluster with snapshots kept in it

  • primary - for a primary cluster which will send snapshots to a backup cluster

  • backup - for a backup cluster which only stores snapshots from other clusters

Note

VolumeCare working in backup mode can have multiple primary clusters sending backups. This is natively supported provided that policies defined in all the primary clusters do not contradict each other and all of them are defined in the backup cluster as well.

Note

For multicluster setups, some modes are currently being worked on.

When the primary mode is selected, a remote location also needs to be specified in the configuration. It will tell the VolumeCare which is the backup cluster that should recieve snapshots from it. This is achieved by setting the remote=<location-name> option. Its value should be the same as a remote location name seen in storpool location list.

Last thing that needs to specified is the driver option. Currently only driver=storpool is supported.

Policy and template sections are described further down in this document.

3. Retention policies

VolumeCare creates and manages snapshot based on retention policies. They are defined in the configuration file in sections with the [policy:<policy-name>] template.

The policy name is defined in the section header.

The essence of each policy is defined by the policy mode, so the mode option should be specified in each policy section. Depending on the policy mode, other options should also be specified in the policy section to provide the policy parameters.

The list of policies for a cluster (or a set of clusters) is immutable, i.e. new policies can be added, but existing ones must not be removed or modified. This is required as otherwise the state of the already created snapshots will be undefined.

Below is the current list of policy modes:

3.1. Local

3.1.1. stopgap

This mode has two parameters - snapshots and interval (in hours). For each entity (volume or virtual machine) with this policy the VolumeCare will keep the specified number of snapshots, each of which is created at the interval specified. For example, 4 snapshots with interval of 6 hours means that the oldest snapshot will be 18-24h old and there will be three more, spaced 6 hours apart.

Example for a virtual machine snapshoted with stopgap, snapshots=5, interval=6:

State ID: 1581687345 -- VM 1000 @ 1581687345 (loc) -- volumes: volume-test2, volume-test9 -- stopgap-5-6 -- 2020-02-14 15:35:45 (0h 18m ago)
State ID: 1581665745 -- VM 1000 @ 1581665745 (loc) -- volumes: volume-test2, volume-test9 -- stopgap-5-6 -- 2020-02-14 09:35:45 (6h 18m ago)
State ID: 1581644145 -- VM 1000 @ 1581644145 (loc) -- volumes: volume-test2, volume-test9 -- stopgap-5-6 -- 2020-02-14 03:35:45 (12h 18m ago)
State ID: 1581622545 -- VM 1000 @ 1581622545 (loc) -- volumes: volume-test2, volume-test9 -- stopgap-5-6 -- 2020-02-13 21:35:45 (18h 18m ago)
State ID: 1581600945 -- VM 1000 @ 1581600945 (loc) -- volumes: volume-test2, volume-test9 -- stopgap-5-6 -- 2020-02-13 15:35:45 (1d 0h 18m ago)

3.1.2. exp

Mode exp has an exponentially increasing interval between the snapshots. Currently it’s parameterless and keeps 12-13 snapshots with the following ages:

  • 4 from the last 3 hours

  • 2-3 aged 3-12 hours

  • 2-3 aged 12-24 hours

  • 4 aged 24-48 hours

Example for a volume snapshoted with exp:

State ID: 1581687584 -- spvc___1581687584___loc___exp-test (loc) -- exp -- 2020-02-14 15:39:44 (0h 14m ago)
State ID: 1581683984 -- spvc___1581683984___loc___exp-test (loc) -- exp -- 2020-02-14 14:39:44 (1h 14m ago)
State ID: 1581680384 -- spvc___1581680384___loc___exp-test (loc) -- exp -- 2020-02-14 13:39:44 (2h 14m ago)
State ID: 1581676784 -- spvc___1581676784___loc___exp-test (loc) -- exp -- 2020-02-14 12:39:44 (3h 14m ago)
State ID: 1581673184 -- spvc___1581673184___loc___exp-test (loc) -- exp -- 2020-02-14 11:39:44 (4h 14m ago)
State ID: 1581665984 -- spvc___1581665984___loc___exp-test (loc) -- exp -- 2020-02-14 09:39:44 (6h 14m ago)
State ID: 1581655184 -- spvc___1581655184___loc___exp-test (loc) -- exp -- 2020-02-14 06:39:44 (9h 14m ago)
State ID: 1581640784 -- spvc___1581640784___loc___exp-test (loc) -- exp -- 2020-02-14 02:39:44 (13h 14m ago)
State ID: 1581629984 -- spvc___1581629984___loc___exp-test (loc) -- exp -- 2020-02-13 23:39:44 (16h 14m ago)
State ID: 1581604784 -- spvc___1581604784___loc___exp-test (loc) -- exp -- 2020-02-13 16:39:44 (23h 14m ago)
State ID: 1581575984 -- spvc___1581575984___loc___exp-test (loc) -- exp -- 2020-02-13 08:39:44 (1d 7h 14m ago)
State ID: 1581543584 -- spvc___1581543584___loc___exp-test (loc) -- exp -- 2020-02-12 23:39:44 (1d 16h 14m ago)

3.1.3. keep-daily

The mode has two parameters - interval and days. It will create snapshots at every interval hours. All snapshots created in the last 24 hours will be kept and snapshots older than 24 hours will be reduced to one per day. All snapshots older than days will be deleted. Essentially, it is the same as stopgap-remote (see below), but all snapshots are kept locally.

Example for a virtual machine snapshoted with keep-daily, interval=1, days=7:

State ID: 1581687584 -- VM 1024 @ 1581687584 (loc) -- volumes: test -- keep-daily -- 2020-02-14 15:39:44 (0h 14m ago)
State ID: 1581683984 -- VM 1024 @ 1581683984 (loc) -- volumes: test -- keep-daily -- 2020-02-14 14:39:44 (1h 14m ago)
State ID: 1581680384 -- VM 1024 @ 1581680384 (loc) -- volumes: test -- keep-daily -- 2020-02-14 13:39:44 (2h 14m ago)
State ID: 1581676784 -- VM 1024 @ 1581676784 (loc) -- volumes: test -- keep-daily -- 2020-02-14 12:39:44 (3h 14m ago)
State ID: 1581673184 -- VM 1024 @ 1581673184 (loc) -- volumes: test -- keep-daily -- 2020-02-14 11:39:44 (4h 14m ago)
State ID: 1581669584 -- VM 1024 @ 1581669584 (loc) -- volumes: test -- keep-daily -- 2020-02-14 10:39:44 (5h 14m ago)
State ID: 1581665984 -- VM 1024 @ 1581665984 (loc) -- volumes: test -- keep-daily -- 2020-02-14 09:39:44 (6h 14m ago)
State ID: 1581662384 -- VM 1024 @ 1581662384 (loc) -- volumes: test -- keep-daily -- 2020-02-14 08:39:44 (7h 14m ago)
State ID: 1581658784 -- VM 1024 @ 1581658784 (loc) -- volumes: test -- keep-daily -- 2020-02-14 07:39:44 (8h 14m ago)
State ID: 1581655184 -- VM 1024 @ 1581655184 (loc) -- volumes: test -- keep-daily -- 2020-02-14 06:39:44 (9h 14m ago)
State ID: 1581651584 -- VM 1024 @ 1581651584 (loc) -- volumes: test -- keep-daily -- 2020-02-14 05:39:44 (10h 14m ago)
State ID: 1581647984 -- VM 1024 @ 1581647984 (loc) -- volumes: test -- keep-daily -- 2020-02-14 04:39:44 (11h 14m ago)
State ID: 1581644384 -- VM 1024 @ 1581644384 (loc) -- volumes: test -- keep-daily -- 2020-02-14 03:39:44 (12h 14m ago)
State ID: 1581640784 -- VM 1024 @ 1581640784 (loc) -- volumes: test -- keep-daily -- 2020-02-14 02:39:44 (13h 14m ago)
State ID: 1581637184 -- VM 1024 @ 1581637184 (loc) -- volumes: test -- keep-daily -- 2020-02-14 01:39:44 (14h 14m ago)
State ID: 1581633584 -- VM 1024 @ 1581633584 (loc) -- volumes: test -- keep-daily -- 2020-02-14 00:39:44 (15h 14m ago)
State ID: 1581629984 -- VM 1024 @ 1581629984 (loc) -- volumes: test -- keep-daily -- 2020-02-13 23:39:44 (16h 14m ago)
State ID: 1581626384 -- VM 1024 @ 1581626384 (loc) -- volumes: test -- keep-daily -- 2020-02-13 22:39:44 (17h 14m ago)
State ID: 1581622784 -- VM 1024 @ 1581622784 (loc) -- volumes: test -- keep-daily -- 2020-02-13 21:39:44 (18h 14m ago)
State ID: 1581619184 -- VM 1024 @ 1581619184 (loc) -- volumes: test -- keep-daily -- 2020-02-13 20:39:44 (19h 14m ago)
State ID: 1581615584 -- VM 1024 @ 1581615584 (loc) -- volumes: test -- keep-daily -- 2020-02-13 19:39:44 (20h 14m ago)
State ID: 1581611984 -- VM 1024 @ 1581611984 (loc) -- volumes: test -- keep-daily -- 2020-02-13 18:39:44 (21h 14m ago)
State ID: 1581608384 -- VM 1024 @ 1581608384 (loc) -- volumes: test -- keep-daily -- 2020-02-13 17:39:44 (22h 14m ago)
State ID: 1581604784 -- VM 1024 @ 1581604784 (loc) -- volumes: test -- keep-daily -- 2020-02-13 16:39:44 (23h 14m ago)
State ID: 1581583184 -- VM 1024 @ 1581583184 (loc) -- volumes: test -- keep-daily -- 2020-02-13 10:39:44 (1d 5h 14m ago)
State ID: 1581496784 -- VM 1024 @ 1581496784 (loc) -- volumes: test -- keep-daily -- 2020-02-12 10:39:44 (2d 5h 14m ago)
State ID: 1581410384 -- VM 1024 @ 1581410384 (loc) -- volumes: test -- keep-daily -- 2020-02-11 10:39:44 (3d 5h 14m ago)
State ID: 1581323984 -- VM 1024 @ 1581323984 (loc) -- volumes: test -- keep-daily -- 2020-02-10 10:39:44 (4d 5h 14m ago)
State ID: 1581237584 -- VM 1024 @ 1581237584 (loc) -- volumes: test -- keep-daily -- 2020-02-09 10:39:44 (5d 5h 14m ago)
State ID: 1581151184 -- VM 1024 @ 1581151184 (loc) -- volumes: test -- keep-daily -- 2020-02-08 10:39:44 (6d 5h 14m ago)
State ID: 1581064784 -- VM 1024 @ 1581064784 (loc) -- volumes: test -- keep-daily -- 2020-02-07 10:39:44 (7d 5h 14m ago)

3.1.4. nosnap

Use this mode for a policy that does not create snapshots. Nosnap policies are usually used in two scenarios - as a default that is overriden per entity, or as a override of another default.

3.2. Remote

3.2.1. stopgap-mirror

This mode is available only in primary/backup mode and has two parameters - interval and snapshots. Essentially it is the same as the local stopgap mode (see above), but copies all snapshots to the backup cluster as well.

Example for a volume snapshoted with stopgap-mirror, interval=24, snapshots=3:

State ID: 1581867697 -- spvc___1581867697___loc___test (loc) -- stopgap-mirror -- 2020-02-16 17:41:37 (16h 27m ago)
State ID: 1581781297 -- spvc___1581781297___loc___test (loc) -- stopgap-mirror -- 2020-02-15 17:41:37 (1d 16h 27m ago)
State ID: 1581694897 -- spvc___1581694897___loc___test (loc) -- stopgap-mirror -- 2020-02-14 17:41:37 (2d 16h 27m ago)
State ID: 1581867697 -- spvc___1581867697___loc2___test (loc2) -- stopgap-mirror -- 2020-02-16 17:41:37 (16h 27m ago)
State ID: 1581781297 -- spvc___1581781297___loc2___test (loc2) -- stopgap-mirror -- 2020-02-15 17:41:37 (1d 16h 27m ago)
State ID: 1581694897 -- spvc___1581694897___loc2___test (loc2) -- stopgap-mirror -- 2020-02-14 17:41:37 (2d 16h 27m ago)

3.2.2. basic-remote

Added in version 1.12. Essentially it follows the same logic as the stopgap and stopgap-mirror modes (see above), but keeps all snapshots except the most recent one in the backup cluster only. The parameters are also the same - interval and snapshots.

Example for a volume snapshoted with basic-remote, interval=24, snapshots=3:

State ID: 1581867697 -- spvc___1581867697___loc___test (loc) -- basic-remote -- 2020-02-16 17:41:37 (16h 27m ago)
State ID: 1581867697 -- spvc___1581867697___loc2___test (loc2) -- basic-remote -- 2020-02-16 17:41:37 (16h 27m ago)
State ID: 1581781297 -- spvc___1581781297___loc2___test (loc2) -- basic-remote -- 2020-02-15 17:41:37 (1d 16h 27m ago)
State ID: 1581694897 -- spvc___1581694897___loc2___test (loc2) -- basic-remote -- 2020-02-14 17:41:37 (2d 16h 27m ago)

3.2.3. stopgap-remote

This mode is available only in primary/backup mode and has two parameters - interval and days. It creates snapshots at every interval hours, and copies all of them in the configured backup cluster. The service will keep all snapshots from the last 24 hours in both clusters. Snapshots older than 24 hours will be reduced to one per day and will be kept in the backup cluster only. Snapshots older than days days will be deleted.

Example for a virtual machine snapshoted with stopgap-remote, interval=1, days=7:

State ID: 1581687584 -- VM 1572 @ 1581687584 (loc) -- volumes: test -- stopgap-remote -- 2020-02-14 15:39:44 (0h 14m ago)
State ID: 1581683984 -- VM 1572 @ 1581683984 (loc) -- volumes: test -- stopgap-remote -- 2020-02-14 14:39:44 (1h 14m ago)
State ID: 1581680384 -- VM 1572 @ 1581680384 (loc) -- volumes: test -- stopgap-remote -- 2020-02-14 13:39:44 (2h 14m ago)
State ID: 1581676784 -- VM 1572 @ 1581676784 (loc) -- volumes: test -- stopgap-remote -- 2020-02-14 12:39:44 (3h 14m ago)
State ID: 1581673184 -- VM 1572 @ 1581673184 (loc) -- volumes: test -- stopgap-remote -- 2020-02-14 11:39:44 (4h 14m ago)
State ID: 1581669584 -- VM 1572 @ 1581669584 (loc) -- volumes: test -- stopgap-remote -- 2020-02-14 10:39:44 (5h 14m ago)
State ID: 1581665984 -- VM 1572 @ 1581665984 (loc) -- volumes: test -- stopgap-remote -- 2020-02-14 09:39:44 (6h 14m ago)
State ID: 1581662384 -- VM 1572 @ 1581662384 (loc) -- volumes: test -- stopgap-remote -- 2020-02-14 08:39:44 (7h 14m ago)
State ID: 1581658784 -- VM 1572 @ 1581658784 (loc) -- volumes: test -- stopgap-remote -- 2020-02-14 07:39:44 (8h 14m ago)
State ID: 1581655184 -- VM 1572 @ 1581655184 (loc) -- volumes: test -- stopgap-remote -- 2020-02-14 06:39:44 (9h 14m ago)
State ID: 1581651584 -- VM 1572 @ 1581651584 (loc) -- volumes: test -- stopgap-remote -- 2020-02-14 05:39:44 (10h 14m ago)
State ID: 1581647984 -- VM 1572 @ 1581647984 (loc) -- volumes: test -- stopgap-remote -- 2020-02-14 04:39:44 (11h 14m ago)
State ID: 1581644384 -- VM 1572 @ 1581644384 (loc) -- volumes: test -- stopgap-remote -- 2020-02-14 03:39:44 (12h 14m ago)
State ID: 1581640784 -- VM 1572 @ 1581640784 (loc) -- volumes: test -- stopgap-remote -- 2020-02-14 02:39:44 (13h 14m ago)
State ID: 1581637184 -- VM 1572 @ 1581637184 (loc) -- volumes: test -- stopgap-remote -- 2020-02-14 01:39:44 (14h 14m ago)
State ID: 1581633584 -- VM 1572 @ 1581633584 (loc) -- volumes: test -- stopgap-remote -- 2020-02-14 00:39:44 (15h 14m ago)
State ID: 1581629984 -- VM 1572 @ 1581629984 (loc) -- volumes: test -- stopgap-remote -- 2020-02-13 23:39:44 (16h 14m ago)
State ID: 1581626384 -- VM 1572 @ 1581626384 (loc) -- volumes: test -- stopgap-remote -- 2020-02-13 22:39:44 (17h 14m ago)
State ID: 1581622784 -- VM 1572 @ 1581622784 (loc) -- volumes: test -- stopgap-remote -- 2020-02-13 21:39:44 (18h 14m ago)
State ID: 1581619184 -- VM 1572 @ 1581619184 (loc) -- volumes: test -- stopgap-remote -- 2020-02-13 20:39:44 (19h 14m ago)
State ID: 1581615584 -- VM 1572 @ 1581615584 (loc) -- volumes: test -- stopgap-remote -- 2020-02-13 19:39:44 (20h 14m ago)
State ID: 1581611984 -- VM 1572 @ 1581611984 (loc) -- volumes: test -- stopgap-remote -- 2020-02-13 18:39:44 (21h 14m ago)
State ID: 1581608384 -- VM 1572 @ 1581608384 (loc) -- volumes: test -- stopgap-remote -- 2020-02-13 17:39:44 (22h 14m ago)
State ID: 1581604784 -- VM 1572 @ 1581604784 (loc) -- volumes: test -- stopgap-remote -- 2020-02-13 16:39:44 (23h 14m ago)
State ID: 1581687584 -- VM 1572 @ 1581687584 (loc2) -- volumes: test -- stopgap-remote -- 2020-02-14 15:39:44 (0h 14m ago)
State ID: 1581683984 -- VM 1572 @ 1581683984 (loc2) -- volumes: test -- stopgap-remote -- 2020-02-14 14:39:44 (1h 14m ago)
State ID: 1581680384 -- VM 1572 @ 1581680384 (loc2) -- volumes: test -- stopgap-remote -- 2020-02-14 13:39:44 (2h 14m ago)
State ID: 1581676784 -- VM 1572 @ 1581676784 (loc2) -- volumes: test -- stopgap-remote -- 2020-02-14 12:39:44 (3h 14m ago)
State ID: 1581673184 -- VM 1572 @ 1581673184 (loc2) -- volumes: test -- stopgap-remote -- 2020-02-14 11:39:44 (4h 14m ago)
State ID: 1581669584 -- VM 1572 @ 1581669584 (loc2) -- volumes: test -- stopgap-remote -- 2020-02-14 10:39:44 (5h 14m ago)
State ID: 1581665984 -- VM 1572 @ 1581665984 (loc2) -- volumes: test -- stopgap-remote -- 2020-02-14 09:39:44 (6h 14m ago)
State ID: 1581662384 -- VM 1572 @ 1581662384 (loc2) -- volumes: test -- stopgap-remote -- 2020-02-14 08:39:44 (7h 14m ago)
State ID: 1581658784 -- VM 1572 @ 1581658784 (loc2) -- volumes: test -- stopgap-remote -- 2020-02-14 07:39:44 (8h 14m ago)
State ID: 1581655184 -- VM 1572 @ 1581655184 (loc2) -- volumes: test -- stopgap-remote -- 2020-02-14 06:39:44 (9h 14m ago)
State ID: 1581651584 -- VM 1572 @ 1581651584 (loc2) -- volumes: test -- stopgap-remote -- 2020-02-14 05:39:44 (10h 14m ago)
State ID: 1581647984 -- VM 1572 @ 1581647984 (loc2) -- volumes: test -- stopgap-remote -- 2020-02-14 04:39:44 (11h 14m ago)
State ID: 1581644384 -- VM 1572 @ 1581644384 (loc2) -- volumes: test -- stopgap-remote -- 2020-02-14 03:39:44 (12h 14m ago)
State ID: 1581640784 -- VM 1572 @ 1581640784 (loc2) -- volumes: test -- stopgap-remote -- 2020-02-14 02:39:44 (13h 14m ago)
State ID: 1581637184 -- VM 1572 @ 1581637184 (loc2) -- volumes: test -- stopgap-remote -- 2020-02-14 01:39:44 (14h 14m ago)
State ID: 1581633584 -- VM 1572 @ 1581633584 (loc2) -- volumes: test -- stopgap-remote -- 2020-02-14 00:39:44 (15h 14m ago)
State ID: 1581629984 -- VM 1572 @ 1581629984 (loc2) -- volumes: test -- stopgap-remote -- 2020-02-13 23:39:44 (16h 14m ago)
State ID: 1581626384 -- VM 1572 @ 1581626384 (loc2) -- volumes: test -- stopgap-remote -- 2020-02-13 22:39:44 (17h 14m ago)
State ID: 1581622784 -- VM 1572 @ 1581622784 (loc2) -- volumes: test -- stopgap-remote -- 2020-02-13 21:39:44 (18h 14m ago)
State ID: 1581619184 -- VM 1572 @ 1581619184 (loc2) -- volumes: test -- stopgap-remote -- 2020-02-13 20:39:44 (19h 14m ago)
State ID: 1581615584 -- VM 1572 @ 1581615584 (loc2) -- volumes: test -- stopgap-remote -- 2020-02-13 19:39:44 (20h 14m ago)
State ID: 1581611984 -- VM 1572 @ 1581611984 (loc2) -- volumes: test -- stopgap-remote -- 2020-02-13 18:39:44 (21h 14m ago)
State ID: 1581608384 -- VM 1572 @ 1581608384 (loc2) -- volumes: test -- stopgap-remote -- 2020-02-13 17:39:44 (22h 14m ago)
State ID: 1581604784 -- VM 1572 @ 1581604784 (loc2) -- volumes: test -- stopgap-remote -- 2020-02-13 16:39:44 (23h 14m ago)
State ID: 1581583184 -- VM 1572 @ 1581583184 (loc2) -- volumes: test -- stopgap-remote -- 2020-02-13 10:39:44 (1d 5h 14m ago)
State ID: 1581496784 -- VM 1572 @ 1581496784 (loc2) -- volumes: test -- stopgap-remote -- 2020-02-12 10:39:44 (2d 5h 14m ago)
State ID: 1581410384 -- VM 1572 @ 1581410384 (loc2) -- volumes: test -- stopgap-remote -- 2020-02-11 10:39:44 (3d 5h 14m ago)
State ID: 1581323984 -- VM 1572 @ 1581323984 (loc2) -- volumes: test -- stopgap-remote -- 2020-02-10 10:39:44 (4d 5h 14m ago)
State ID: 1581237584 -- VM 1572 @ 1581237584 (loc2) -- volumes: test -- stopgap-remote -- 2020-02-09 10:39:44 (5d 5h 14m ago)
State ID: 1581151184 -- VM 1572 @ 1581151184 (loc2) -- volumes: test -- stopgap-remote -- 2020-02-08 10:39:44 (6d 5h 14m ago)
State ID: 1581064784 -- VM 1572 @ 1581064784 (loc2) -- volumes: test -- stopgap-remote -- 2020-02-07 10:39:44 (7d 5h 14m ago)

4. Policy resolution

Each entity (volume or virtual machine) should have a policy applied to it. The policies are applied to, in order of precedence:

  • volumes, by adding a tag vc-policy=<policy-name> to the volume;

  • templates, via [template:<template-name>] section in the config file with the policy=<policy-name> option in it;

  • the whole cluster, via the [template:*] section. This section is mandatory.

Virtual Machines have the common policy of all their volumes. All volumes of a virtual machine must have the same retention policy, otherwise it is considered a misconfiguration and the virtual machine will be ignored by the service.

Note

[template:*] is mandatory, otherwise VolumeCare will refuse to start.

5. storpool_vcctl

Note

Initially storpool_vcctl is in /root/storpool folder.

To control VolumeCare, StorPool provides the storpool_vcctl tool. It’s usage is:

usage: storpool_vcctl [-h] [-C CONFIGFILE] {show,list,status,revert} ...

The following sub-commands are available:

5.1. show

usage: storpool_vcctl show vm VMTAG=VMID
       storpool_vcctl show volume VOLUME

This sub-command shows the snapshots created for a volume or virtual machine. Below are two examples:

[root@vc volumecare]# ./storpool_vcctl.py show volume test
volume: test (loc) -- stopgap-short
   State ID: 1581928784 -- spvc___1581928784___loc___test (loc) -- stopgap-short -- 2020-02-17 10:39:44 (0h 12m ago)
   State ID: 1581925184 -- spvc___1581925184___loc___test (loc) -- stopgap-short -- 2020-02-17 09:39:44 (1h 12m ago)
   State ID: 1581921584 -- spvc___1581921584___loc___test (loc) -- stopgap-short -- 2020-02-17 08:39:44 (2h 12m ago)
test (loc) total: 3; local: 3; remote: 0

Here you can see three snapshots for a single volume, 1 hour apart.

[root@vc volumecare]# ./storpool_vcctl.py show vm nvm=2
vm: nvm=2 (loc) -- stopgap-short
 State ID: 1581928784 -- NVM 2 @ 1581928784 (loc) -- volumes: test5, test6, test7, test8 -- stopgap-short -- 2020-02-17 10:39:44 (0h 9m ago)
 State ID: 1581925184 -- NVM 2 @ 1581925184 (loc) -- volumes: test5, test6, test7, test8 -- stopgap-short -- 2020-02-17 09:39:44 (1h 9m ago)
 State ID: 1581921584 -- NVM 2 @ 1581921584 (loc) -- volumes: test5, test6, test7, test8 -- stopgap-short -- 2020-02-17 08:39:44 (2h 9m ago)
nvm=2 (loc) total: 3; local: 3; remote: 0

In this example, the orchestration uses the nvm tag to mark the volumes belonging to a virtual machine. The status shows that there are 4 drives on the VM, and they have 3 previous snapshots, 1 hour apart.

5.2. list

usage: storpool_vcctl list {volumes,vms,policies}

This sub-command lists the affected volumes, VMs and active policies. Examples:

[root@vc volumecare]# ./storpool_vcctl.py list vms
cvm=1 (loc)
cvm=1024 (loc)
nvm=1 (loc)
nvm=2 (loc)

A setup of three VMs, two identified with nvm tag and one with a cvm tag.

[root@vc volumecare]# ./storpool_vcctl.py list volumes
bridge-test (loc)
company-test (loc)
one-img-9 (loc)
35f12f61-5c43-48f1-8036-3b3c254e8a54 (loc)
one-img-93 (loc)
one-img-94 (loc)
one-img-95 (loc)
one-img-96 (loc)
one-img-97 (loc)
one-img-98 (loc)
8a293540-33fc-4f33-8aa2-a761cbe0684e (loc)

This is a short list of the standalone volumes the service takes care of.

[root@vc volumecare]# ./storpool_vcctl.py list policies

[policy:stopgap-short]
mode=stopgap
snapshots=3
interval=1

[policy:cust-main]
mode=exp

[policy:no]
mode=nosnap

[policy:keep-daily]
mode=keep-daily
interval=2
days=7

5.3. status

This sub-command is the same as show for all VMs and volumes.

5.4. revert

Warning

PLEASE MAKE SURE you have read and understood the implications before using this command.

usage: storpool_vcctl.py revert [-h] {rename,delete} {volume,vm} ...
       storpool_vcctl.py revert {rename,delete} vm [-N] vm_id state_id
       storpool_vcctl.py revert {rename,delete} volume [-N] volume_id state_id

This sub-command reverts a specified volume or VM to a previous state.

The following constraints exist:

  • The volumes must not be attached to any node

The first argument (rename or delete) specifies what will be done to the existing volumes that will be reverted:

  • rename will rename them to VOLUMENAME_dead_TIMESTAMP, where TIMESTAMP is the current Unix time.

  • delete will delete them.

A state is defined by an Unix timestamp and can be obtained via the list sub-command.

The -N option will not do any changes, but will report what will be done. It is strongly recommended to run every command first with -N , to make sure it does what is expected.

6. Changing a policy

The policies are best explained as immutable objects, thus when a change is required, the old policy needs to be kept until the new policy is applied. This immutability is not enforced due to rare exceptions requiring actual changes in an existing policy, but is strongly encouraged.

6.1. Single-cluster

Policy changes are best performed by creating a new policy and leaving the snapshots in the old one to expire when this is possible. Any changes in the /etc/storpool/volumecare.conf require restarting the storpool_volumecare service (presently running in the first node in a cluster if configured).

6.2. Multiple clusters

Changing a remote policy requires the change to occur both in the local and the remote clusters, otherwise the policy goes out of sync. A monitoring alert for policies out of sync is in development and will alert for such issues once done.

7. Example configurations

7.1. Single-cluster

[format]
version=1.0

[volumecare]
mode=normal
driver=storpool

[policy:stopgap-short]
mode=stopgap
snapshots=3
interval=1

[policy:cust-main]
mode=exp

[policy:no]
mode=nosnap

[policy:keep-daily]
mode=keep-daily
interval=2
days=7

[template:one-ds-0]
policy=stopgap-short

[template:one-ds-1]
policy=cust-main

[template:*]
policy=no

7.2. Primary cluster

[format]
version=1.0

[volumecare]
mode=primary
driver=storpool
remote=cust-tier2

[policy:cust-main-remote]
mode=stopgap-remote
interval=2
days=7

[policy:cust-main]
mode=keep-daily
interval=2
days=7

[policy:no]
mode=nosnap

[template:one-ds-0]
policy=cust-main-remote

[template:one-ds-1]
policy=cust-main

[template:*]
policy=no

7.3. Backup cluster

[format]
version=1.0

[volumecare]
mode=backup
driver=storpool

[policy:cust-main-remote]
mode=stopgap-remote
interval=2
days=7

[policy:cust-main]
mode=keep-daily
interval=2
days=7

[policy:no]
mode=nosnap

[template:*]
policy=no

8. Changelog

1.15

  • Small bugfix of the CARE task rescheduling with the same timestamp

1.14

  • vcctl: add json output for status

  • vcctl: add option to hide entities with policy nosnap

  • vcctl: show vms when searching for volume

  • vcctl: add local and remote filtering to show and status commands

  • Implement a timeout for the obsolete snapshots check

  • vcctl: add internal storpool snapshot stats to status

  • Add a jq module for transfers in the primary clusters

1.13

  • Do not unexport in the deletion tasks in primary clusters

  • Add the inherit_tags functionality

1.12

  • Added basic-remote policy

  • Fixed some bugs in stopgap-mirror

  • vcctl: added verbose output; vm storpool snapshot names can be seen there

  • vcctl: implemented show policy

  • vcctl: show and list accept location as well

1.11

  • Do not add export/unexport tasks for snapshots that are pending deletion.

1.10

  • Fixed the location deducing of some volumes.

1.09

  • Backup clusters got “next_remote” for exporting snapshots to one more place.

1.08

  • Fixed a race condition bug on transfer/delete snapshot in primary clusters.

  • stopgap-remote now deleted the non-daily snapshots form the primary if they expire (got age > 24h) immediately, not keeping them to be transferred to the backup.

1.07

  • Track snapshot exports and unexports with events.

  • Protection for duplicating delete/export/unexport/copy tasks.

  • Use only location ids internally, remote option in the config is not affected.

  • Don’t show recovering snapshots from the storpool driver. It is conceptually wrong. A snapshot is present when it is recovered.

  • Backup clusters now manually export snapshots after recovering. This is needed so that vm volumes snapshots appear at the same time.

  • Default retry policy changed to: initial - 3 min, increment - x2.

1.06

  • VolumeCare in normal mode does not scan for remote snapshots.

1.05

  • Primary clusters do the exports per entity one by one.

  • Backup clusters export special dummy snapshot to the primary for location tracking. Primary clusters do not export snapshots to down locations.

1.04

  • Reschedule the care task immediately if there are unhandled events.

  • Primary clusters unexport snapshots after they are transferred.

1.03

  • Add stopgap-mirror policy.

  • Fix the issue with timeutils.now that could cause 15s premature task exection

1.02

  • Add -L/–location option to vcctl status; querries volumes/vms only from the given location. Locations are raw (e.g. bbht - the first part of CLUSTER_ID).

  • Volumecare in primary mode now exports the snapshots sorted by age, oldest first (in the same manner the volumecare in backup mode invokes copyFromRemote).

  • Volumecare in backup mode now does not require the “remote” option in the config. Backup mode natively supports multiple primary clusters and that option was actually never used when running in backup mode.

1.01

  • Fix vcctl show volume <volume_name> not working.

1.0

  • initial