StorPool 19.1 release notes

19.1 revision 19.01.1628.1b627d0

Released: 07 Dec 2020

Now collecting additional configuration data in storpool_stat for additional per-host monitoring checks, includes lldpd as dependency.
Adds Volumecare version 1.18 that includes support for multiple sub-clusters sending to a backup location.
Includes improvements and and fixes in internal tools, core and services.
Adds compression and proxy authorization configuration support options for the monitoring collection scripts.
Adds the SP_MIRROR_DIR option for containerized workloads.
Adds support for the Micron 9300 MTFDHAL3T2TDR NVMe drive.
Adds compression support for monitoring data collection scripts.

19.1 revision 19.01.1548.00e5a5633

Released: 27 Nov 2020

Adds fix for storpool_stat for handling statistics for unnamed volumes.
Adds support for Samsung PM173X NVMe drives.
Adds fix for iSCSI re-assign targets implementation to retry on transaction reset.
Improves storpool_stat send performance after longer periods of no connectivity
Adds storpool_initidks –no-notify option for easier StorPool disk attributes changes.
Adds reliability improvements in the in-server disk tester implementation
Adds an additional iSCSI helper option to the iface-genconf tool (see 8.2.8. iSCSI configuration)

The main reason for the extension is a planned change in the monitoring system to periodically check if the portal group and portal addresses exposed by the storpool_iscsi controller services are still reachable. This change will require adding OS/kernel interfaces for each of the portal groups, so that they could be used as a source IP endpoint for the periodic monitoring checks.
Adds fix for disk not getting ejected by the server after a drive was re-balanced out.
Adds workaround for a bug in the i40e driver in CentOS6 based NICs (hardware acceleration)

19.1 revision 19.01.1511.0b533fb

Released: 28 Oct 2020

Adds fixes for idle service VF resets with ixgbe driver based NICs.
Adds fixes for volumecare tags handling with unnamed volumes.
Adds initial support for CentOS 8.2 installations.
Adds tools for setting up volume overrides for many volumes from the same parent use case. For details, see Volume overrides.

Presently, overrides are manually added by StorPool support only in places that require them. At a later point they will be re-created periodically, so that new volume objects are included in this analysis regularly.
Increases the listen backlog for the storpool_mgmt and storpool_bridge services.

19.1 revision 19.01.1468.90a9873

Released: 16 Oct 2020

Adds support for Micron 9300 NVMe.
Adds workaround for firmware issues with bnxt_en NICs (detect and reset stuck VF).
Improves handling of LACP mode i40e based NICs.
Fixes an issue with handling old-style iSCSI reservations used by VMWare 4.x and 5.x.
Fixes aborts in storpool_iscsi with bnxt_en driver when a large ICMP packet is received.
Improves StorPool API operation with large number of volumes and snapshots with multicluster environment.
Adds more granular flags for snapshot dematerialization status.
Adds fixes for iSCSI sessions list (known as empty target/initiator in CLI).
Adds fix for CLI not showing unnamed volumes name in attachments list.
Removes deprecated options from storpool.conf.example.
Includes updates and fixes in internal tools.

19.1 revision 19.01.1413.b807f92

Released: 26 Sep 2020

Updated dependencies when MLNX OFED is used.
Balancer fix for cases where overrides are enabled.
Adds bridge multicluster stability fixes.
Updates Python bindings with missing functionality available in the present API.
Adds diagnostic features for the /usr/lib/storpool/monitor utility.
Adds handling for misbehaving i40e based NICs that could lead to aborted services.
Adds updates and fixes in internal tools.

19.1 revision 19.01.1376.ce07826

Released: 21 Aug 2020

Adds volumecare support for unnamed volumes.
The in-server disk tester now sends FUA writes during tests (except for –no-fua y disks).
Improved performance when relocating data from SSD or NVMe disks
Adds handling in api-ip for some obscure kernel 4.18 behavior leaving leftover routes.
Adds a fix for API to not show deleted snapshots addressed by global ID.
Adds a fix for volumeMoveRemote API call now properly handling IOPS and bandwidth limits.
Includes updates and fixes in internal tools.

19.1 revision 19.01.1357.39c014c

Released: 12 Aug 2020

Fixes a bug in handling old-style iSCSI reservations used by VMWare 4.x and 5.x
Adds storpool_volumecare as an installable module (More on storpool_volumecare here)
Adds support for Micron 7300 Pro NVMe
Updates and improvements in internal tools

19.1 revision 19.01.1346.dd68fa2c6

Released: 04 Aug 2020

Cluster volume/snapshot limit bumped to 60k volumes/snapshots.
StorPool API now automatically fails over in case its address vanishes for any reason.
The storpool_cg tool now properly handles disks with custom objects and/or entries counts.
Now automatically patching /etc/kdump.conf to blacklist non-important but memory intensive drivers.
New default TCP sysctl settings for better throughput performance for bridge enabled nodes.
Updated installation helper scripts handling better latest rpm/deb packages and other improvements.

19.1 revision 19.01.1293.cfcb869

Released: 11 Jul 2020

Improvements in storpool_initdisk tool - exit code 0 on --nvme-smart, added --json output
Adds spopenstack in the main source tree
Initial CentOS 8 build support
Major installation package system changes - new implementation of most core installation tools, all service files are now handled as rpm/deb packages
Critical fix for all systems running with hard disk drives, resiliency on power outages.

19.1 revision 19.01.1217.1635af7

Released: 02 Jul 2020

Improves handling of lost packets in iSCSI connections
Adds full cluster maintenance mode monitoring improvements
Server now automatically tests and returns disks on transient failures. This improves the way storage media and/or controller failures are handled.
Improvement in multicluster - volume create/update flags now mandatory for MultiCluster API endpoint, adds VolumeRevert operation
Improvement in multicluster - volume exports and remote volumes can now be listed
Performance improvements - more efficient bridge and API interoperability
Performance improvements in dematerialization
Adds CLI improvements - iSCSI list interfaces
Updates cgtool defaults - the system.slice is now with 4G limit by default
Adds minor fixes and improvements in balancer, build and automation tools

19.1 revision 19.01.1108.02703b8c5

Released: 04 Jun 2020

storpool_bridge performance improvements
Oracle Linux support
Kernel 5.4 support
Increase limit of total volumes and snapshots to 45000
Better support for LACP hashing handling

19.1 revision 19.01.1025.0baac06a6

Released: 04 Jun 2020

Added Maintenance mode
Support for volume creation with global IDs instead of volume names.

StorPool now can allow the creation of a volume without a name provided, by assigning it a globally unique ID, by which it could be then referred.

Note

Globally-unique in this regard means world-wide global uniqueness, i.e. no two StorPool deployments can generate the same ID.

This feature was developed to handle some race conditions and retry-type cases in orchestration systems in multi-cluster environment, which could lead to duplication and reuse of another volume. By allowing the storage system to ensure the uniqueness of volume identifiers, all these cases are solved.
Revert to snapshot functionality
NIC initialization and configuration is now non-blocking.

Non-blocking NIC initialization and configuration allows for StorPool services to continue operating normally during hardware reconfigurations or errors. For example: link flaps, changing MTU or other NIC properties, adding/removing VFs to enable/disable hardware acceleration, and so on.

This was developed to handle delays from hardware-related reconfiguration or initialization issues, as most NICs require resets, recreation of filters, or other tasks which take time for the NIC to process, and if just busy-waited would not allow the services to process other requests in the mean time.

The feature works by sending requests to the NIC for specific changes and rechecking their progress periodically (between processing other tasks), or when waiting for other events to complete.
Availability of information for active iSCSI sessions
Acceleration support for virtio-net based adapters
Added support for global active operations list.

StorPool has support for listing the active requests on disks and clients. With this change the functionality is expanded to be able to get all such data from other services (for example, storpool_bridge and storpool_iscsi) and to show some extra details. This is also expanded to be done with a single call for the whole cluster, greatly simplifying monitoring & debugging.

This feature was developed to replace the latthreshold tool that showed all requests of clients and disks that were taking more than a set time to complete. The present implementation gets all required data without the need for sending a separate API call for each client/disk in the cluster.
storpool_initdisk now has extra validation of flags and devices
reuseServer implicit on a down disk in small clusters.

This feature allows a volume to be created even if a cluster is on the minimum system requirements of three nodes, with a single disk missing from the cluster, or if one of the nodes is down at the moment.

Before this change all attempts to create a volume in this state would have resulted in “Not enough servers or fault sets for replication 3” error for volumes with replication 3.

With this feature enabled the volume will be created as if the volume was created with reuseServer. For details, see 12.22.6. Reusing server on disk failure.

19.1 revision 19.01.878.7b1f83e3d

Released: 04 Jun 2020

Infrastructure to allow customers to access StorPool packages
Tools to help with installing new modules and services
Add bridge buffer size options (location update command), see also Bridge throughput performance
Fix for sending monitoring data in blackholed IPv6 networks

19.1 revision 19.01.759.024d1bd

Released: 18 Mar 2020

iSCSI TCP fixes and optimizations
Improvements in handling of I/O errors for NVMe devices
Compatibility improvement in Python bindings: fix the type of the allowRemoteExported field so that it’s not always sent to the StorPool API

19.1 revision 19.01.742.47c6e9c

Released: 06 Mar 2020

Major improvement in aggregation performance with a significantly more efficient internal trigger
Improvements in dematerialization with mixed server and controller versions in the same cluster
Various fixes and improvements in the in-iSCSI TCP implementation
Performance improvements in the storpool_controller service
Fix for thread priorities in some services that were leading to timeouts and service restarts

19.1 revision 19.01.719.c40dd8c

Released: 26 Feb 2020

Various improvements/fixes in multicluster, for details see Introduction to multi-cluster mode
Various iSCSI fixes and optimizations
A better handling of TRIMs over iSCSI
Improvements in balancer tools, handling some edge cases
Various other improvements in internal tools and services

19.1 revision 19.01.544.39e62dbad

Released: 13 Jan 2020

Adds setirqaff validation option
Adds a required default kernel configuration option for improved performance with known supported storage controllers
Improved storpool_bridge performance under constrained CPU and network bandwidth conditions
Adds colorless output for the storpool_tree utility with flags for volumes and snapshots

19.1 revision 19.01.496.6d2c5bf83

Released: 03 Dec 2019

Removes excessive debug print logging
Improved kdump setup tooling for Debian based systems
Support for larger set of switch hash functions for LACP and ECMP
Updates in fio testing framework scripts, closely following latest monitoring alerting

19.1 revision 19.01.385.fc63315ef

Released: 19 Nov 2019

Adds support for monitoring through IPv6 only hosts.
Adds hardware acceleration support for BCM57412 NetXtreme-E 10Gb Ethernet Controllers.
Adds improved stability of quorum keeping service in unstable network conditions.
Adds stability improvements in the quorum keeping service in unstable networking conditions.
Updates the detection for hardware accelerated NICs for the bnxt_en driver in the cgroup configuration helper tools.
Adds an API fix and improvements related to multi-cluster support with CloudStack deployments; for details about this mode, see Introduction to multi-cluster mode
Adds expansion support for Kubernetes integration.

19.1 revision 19.01.375.684a0da12

Released: 07 Nov 2019

Adds fix for the CLI when a volume is created without a template.
Improvements in logging, handles unexpected verbosity.
Improvements in statistics collection daemons, safer methods for data collection.
Improvements for stability and edge cases handling in API.
Improvements in storpool_cg for converged nodes, single-thread hosts migration fixes.
Improvements in monitoring collection scripts, now collecting and handling more data.

19.1 revision 19.01.355.896c5ebaf

Released: 23 Oct 2019

Improvements in API performance when balancer overrides are in place.
Improvements for iSCSI when persistent reservations (MicroSoft CSV) are being used with target failover.
Improvements for aggregation logic during recovery.
Adds change in CLI for template propagate, now stating propagate is mandatory on template changes.
Adds fix for removing the reuseServer flag for volumes/snapshots previously applied with reuseServer
Adds an automatic disabling through vf-udevhelper for the internal i40e based NICs LLDP listener.
Adds an update storpool_daemon with systemd, now no longer forking and other smaller improvements.
Adds more variables reported by the server_stat tool, visible also in the collected statistics for the cluster.
Adds stability fixes in the storpool_iscsi service.
Adds performance fixes with improved read latency for HDD pools.
Adds default configuration changes for internal command output timeouts when collecting periodic reports.
Improvements in storpool_cg for server cache autodetection during cgroup config migration, various other improvements.

19.1 revision 19.01.318.daa3c5938

Released: 07 Oct 2019

Adds CLI support for multicluster mode, for details see Introduction to multi-cluster mode
Improvements for iSCSI with LACP bonds, unreliable network, etc.
Adds fixes in Python bindings
Improvements in cgtool chain
Improvements in kubernetes integration code
Improvements in API responsiveness during dematerialization

19.1 revision 19.01.301.f1b25e7

Released: 24 Sep 2019

Enables setting the SMP affinity of MegaRAID controllers
Adds support for CentOS 7.7 (kernel 3.10.0-1062.1.1.el7.x86_64)
Adds support for mixed routed/switched networks for iSCSI
Adds various storpool_iscsi improvements and fixes
Various other improvements in internal tools and services

19.1 revision 19.01.271.de5921845

Released: 12 Sep 2019

Improvements in the irq pinning tools for better CPU congestion avoidance on storage service CPU cores
Improvements in internal tools for stress testing drives
Adds support for non-jumbo frames routed networks
Adds support for hardware or no sleep options for bridge service and cgroup configuration tools
Adds support for upgrading 18.02 clusters with iSCSI to accelerated storpool_iscsi
Adds various storpool_iscsi improvements and fixes
Adds initial support for storpool_iscsi persistent reservations

19.1 revision 19.01.212.66fed3091

Released: 10 Sep 2019

Adds support for 4.18 line of kernels
Improvements in python-bindings, monitoring services, cgroup tools
Various other improvements in internal tools and services

19.1 revision 19.01.199.b98e9d4

Released: 29 Aug 2019

Adds priority based recovery for better handling of remote bridge recovery operations
Adds tools for easier upgrade from 18.02 with multi site setup
Adds common configuration for all supported hardware accelerated NIC drivers
Improvements in the tooling for enabling required hardware acceleration settings

19.1 revision 19.01.182.666093099

Adds support for multiple clusters in the same location
Read operations now can be forwarded through the active bridge connection for not fully recovered snapshots
Adds support for 4.20 and 5.0 kernel line
Adds significant protocol improvements for small block size and low IO queue depth read/write operations latency
Adds significant performance improvements for storpool_iscsi, now running only on adapters with support for hardware acceleration (i40e, ixgbe, mlx4_ib, mlx5_ib drivers)
Adds support for iSCSI on layer 3 only networks (routed networks)
Adds support for hardware accelerated bnxt driver based Broadcom adapters
Adds backwards compatibility with pre 19.01 release versions
Fixes AMD EPYC v.1 hardware sleep latency issues