Erasure coding
Is Erasure Coding enabled for a whole cluster, or is it enabled on a per-volume basis?
When erasure coding (EC) is enabled in a cluster, the conversion from triple replication to an erasure coding redundancy scheme is done one volume-snapshot chain at a time. Volume-snapshot chains are converted individually, and the conversion is performed live, so all user operations continue to be processed while the conversion to erasure coding is being performed.
Conversion here means that at least one snapshot is on an EC template and is being encoded. Data in volumes remains triple replicated even when erasure coding is enabled.
Are there any expected performance issues that need to be considered?
During normal operations:
Read and write performance is the same as you get from triple replication - same latency, max IOPS, and bandwidth.
Initial conversion:
Based on our testing, there might be a slight latency increase during the initial conversion from triple replication to the erasure-coded state. In environments where EC is deployed everything is running okay, but just in case we have introduced some configurable conversion speed limits to ensure no latency issues for user input/output operations.
Fault state:
There’s some read performance penalty when you have one or two nodes down. The impact of one node down is about 6 to 10 microseconds increased average read latency. In the case of two nodes down, the performance impact is 20 to 30 microseconds increased average read latency compared to normal operations. In both cases, there is no impact on the write operations.
What hardware configurations are supported?
Initially, erasure coding in StorPool will be supported only for all-NVMe SSD deployments.
What are the supported erasure coding schemes and their respective overheads?
The supported erasure coding schemes are:
2+2 - supports clusters with 5 nodes or more, has approximately 2.4x overhead
4+2 - supports clusters with 7 nodes or more, has approximately 1.8x overhead
8+2 - supports clusters with 11 nodes or more, has approximately 1.5x overhead
Do the required regular snapshots offset some of the space savings?
Each snapshot keeps the difference between the previous snapshot and the volume, so the overhead of each snapshot is proportional to the amount of writing that happened in the volume since the previous snapshot. The total snapshot overhead depends on the number of volumes, the frequency of snapshots, the change rate between each snapshot, and the retention settings of the snapshots.
As a result of the total snapshot overhead, the cluster will have slightly more stored data, but that increase should be negligible compared to the space savings achieved thanks to erasure coding. If needed, we can help you calculate the estimated total snapshot overhead in case you go with the default policy that keeps 24 hourly snapshots and 3 daily snapshots.
Would I need additional space from triple replication to erasure coding during the conversion?
No. There is no point in time when additional space is required for erasure coding compared to triple replication. The data on the drives will be read once to calculate parity blocks and then a rebalance is initiated. As a result, converting data to erasure coding will take less space on drives.
Is there an ideal minimum volume size?
No. Every volume in the system can benefit from the space-saving effect of erasure coding.
How frequently does StorPool recalculate the parity blocks - with each change of the data blocks, or with another method?
StorPool does not recalculate parity blocks. The calculation of parity blocks is done as a background task after creating each regular snapshot of an EC-enabled volume. Since StorPool stores new data in volumes and their unique new data is kept in each newly-created snapshot, there is no need to recalculate older parity blocks.
As older snapshots expire, the common parts between the expired and new snapshots are merged into newer snapshots as a background task.
What is the impact of erasure coding on the network load?
During initial conversion there is a slight increase in network traffic between nodes, which depends on the configured bandwidth limit for the 3N to EC conversion task. During normal operations with EC there’s an extra read of the data to perform the encoding of snapshots (when new snapshots are created).
Is there a difference between the performance of the different erasure coding schemes?
During normal operations:
No, the performance is consistent between all erasure coding schemes.
During initial conversion:
No, the slight latency increase is observed for all erasure coding schemes.
During fault states:
Yes, for 8+2 the impact to latency is slightly higher compared to 4+2 or 2+2 for both 1-node-down and 2-nodes-down events. The numbers under question number 2 are for the 8+2 scheme, so you can expect less performance impact if you use another scheme.
Is there any delay in user operations when regular snapshots are being created?
No. StorPool performs snapshots instantly and with no performance impact.
What are the chances that erasure coding can’t recover from a disk failure?
The level of protection erasure coding provides is equivalent to the protection of triple replication. It can recover from any double failure (that is, failure of any number of storage devices in up to two fault sets). For example, an erasure coded cluster can fully recover if any of the following occurs:
Any two fault sets in the cluster are permanently lost
One fault set and one or more drives in another fault set fail
Multiple drives on two different fault sets fail
Note
When no fault sets are configured, a single node is a fault set.