Volume Ops Weights

This document describes how StorPool write and trim weights work with specific use cases and fio examples.

StorPool version 21.0 revision 21.0.3304.32887b33b introduces four performance settings:

1. writeIopsWeight

2. writeBwWeight

3. trimIopsWeight

4. trimBwWeight

The range is 0.001 (the operation is counted as 1/1000 of a read operation) to 1000 (the operation is counted as 1000 read operations). Trim operations can be set to 0 (unlimited).

The settings are stored in the internal management configuration, are cluster-global, and apply only to volumes that have IOPS or bandwidth limits set.

Use Cases

One use case is absorbing large trims on first boot of new virtual machines on StorPool volumes with configured IOPS or bandwidth limits. At this point the trim operations against the StorPool volume can hit the IOPS or bandwidth limit, and the VM will stall for a while until the trim operations complete. If trimIopsWeight / trimBwWeight is set to 0, or a very low value, the trims will pass faster and the VM will be operational sooner.

Another use case is limiting writes to increase the lifespan of the underlying storage devices.

Walkthrough for write weights with fio

Create a volume called test-ops-weights with size 128G and IOPS 1000, and attach it on the current host:

storpool volume test-ops-weights size 128G iops 1000 template all-flash
storpool attach volume test-ops-weights here

Save the following fio jobs as test-ops-weights.fio:

[global]
ioengine=libaio
direct=1
bs=4k
time_based=1
runtime=10000
clocksource=cpu
filename=/dev/storpool/test-ops-weights

[reads]
rw=randread
rate_iops=500

[writes]
rw=randwrite

and start a fio run with them: fio ./test-ops-weights.fio.

Each job is a separate process that sends either read or write requests to a random offset of the StorPool volume. iodepth=32 means that each process sends 32 requests, waits for at least one request to complete, and then sends more requests to get to 32 requests in flight. The first fio job is issuing 500 randread IOPS, while the second will issue as much randwrite IOPS as possible.

While the jobs are running, notice that the read and write IOPS are roughly equivalent (500/500). With the default configuration, this means:

1000 total StorPool volume iops - 500 randread iops = 500 iops

left for randwrite.

In another terminal, set writeIopsWeight to 0.5:

storpool mgmtConfig writeIopsWeight 0.5

Then, back on the terminal that is running the fio, notice that the split is now 500/1000. Since now each write operation costs half of a read operation (and the reads are fio-capped at 500), the fio get get twice as many writes through.

In the second terminal, set writeIopsWeight to 2:

storpool mgmtConfig writeIopsWeight 2

Back on the fio terminal, the split is now ~ 333/333. Reads and writes are competing for the same total IOPS limit and are running at the same time, roughly equally distributed across each second. But since the writes are now 2 times heavier than reads, the total IOPS limit is reached at 333 + 333*2 ~= 1000.

Reset writeIopsWeight to 1. Stop the fio job, remove the rate_iops=500 from the reads job and start fio again.

Reads and writes should be running with the initial 500/500 split.

In another terminal, set writeIopsWeight to 0.5:

storpool mgmtConfig writeIopsWeight 0.5

The fio should now report ~ 666/666 split. Since the reads are now not fio-capped at 500, both the reads and writes compete for as many IOPS as possible at the same time. As each write operation costs half of a read, the total StorPool volume IOPS limit is filled with 666 + 666*0.5 ~= 1000. When setting writeIopsWeight to 0.001, fio can push 1000/~970 operations through.

On the other end, setting writeIopsWeight to 1000, makes both reads and writes halt to ~1 IOPS.

Bandwidth limits behave the same way.

Trim weights

Currently, trim operations across almost all Linux kernels and fio IO engines are synchronous. If the iodepth=32 of the fio in the previous section is used and just randwrite is replaced with randtrim, fio will issue 32 times more randread requests than trims. To avoid this, set the iodepth=1 when testing trim weights.

Trim results follow the same logic as the write weights in the previous section.

In the unlimited randread rate_iops and trim (in this case iops) weight = 0, instead of getting a 1000/10k (randread/randtrim) IOPS, fio reports ~ 1000/1800. Trim operations are not completely unlimited, they are just not accounted for in the ops budget for next millisecond. They will be not be shaped, but they will still have to wait if there are other operations in the queue.

Summary

The following table summarizes how ops weights work in a two-job randread-randwrite, iodepth=32 fio workload example.

randread rate_iops	writeIopsWeight	Achieved IOPS (randread/randwrite)
500	1	500/500
500	0.5	500/1000
500	2	333/333
500	0.001	500/100K
unlimited	1	500/500
unlimited	0.5	666/666
unlimited	2	333/333
unlimited	0.001	1000/970

The following table summarizes how ops weights work in a two-job randread-randtrim, iodepth=1 fio workload example.

randread rate_iops	trimIopsWeight	Achieved IOPS (randread/randtrim)
500	1	500/500
500	0.5	500/1000
500	2	333/333
500	0.001	500/10K
unlimited	1	500/500
unlimited	0.5	666/666
unlimited	2	333/333
unlimited	0.001	1000/890
unlimited	0	1000/1800