StorPool 19.1 release notes

19.1 revision 19.01.1628.1b627d0

Released: 07 Dec 2020

  • Now collecting additional configuration data in storpool_stat for additional per-host monitoring checks, includes lldpd as dependency.

  • Adds Volumecare version 1.18 that includes support for multiple sub-clusters sending to a backup location.

  • Includes improvements and and fixes in internal tools, core and services.

  • Adds compression and proxy authorization configuration support options for the monitoring collection scripts.

  • Adds the SP_MIRROR_DIR option for containerized workloads.

  • Adds support for the Micron 9300 MTFDHAL3T2TDR NVMe drive.

  • Adds compression support for monitoring data collection scripts.

19.1 revision 19.01.1548.00e5a5633

Released: 27 Nov 2020

  • Adds fix for storpool_stat for handling statistics for unnamed volumes.

  • Adds support for Samsung PM173X NVMe drives.

  • Adds fix for iSCSI re-assign targets implementation to retry on transaction reset.

  • Improves storpool_stat send performance after longer periods of no connectivity

  • Adds storpool_initidks –no-notify option for easier StorPool disk attributes changes.

  • Adds reliability improvements in the in-server disk tester implementation

  • Adds an additional iSCSI helper option to the iface-genconf tool (see 8.2.8.  iSCSI configuration)

    The main reason for the extension is a planned change in the monitoring system to periodically check if the portal group and portal addresses exposed by the storpool_iscsi controller services are still reachable. This change will require adding OS/kernel interfaces for each of the portal groups, so that they could be used as a source IP endpoint for the periodic monitoring checks.

  • Adds fix for disk not getting ejected by the server after a drive was re-balanced out.

  • Adds workaround for a bug in the i40e driver in CentOS6 based NICs (hardware acceleration)

19.1 revision 19.01.1511.0b533fb

Released: 28 Oct 2020

  • Adds fixes for idle service VF resets with ixgbe driver based NICs.

  • Adds fixes for volumecare tags handling with unnamed volumes.

  • Adds initial support for CentOS 8.2 installations.

  • Adds tools for setting up volume overrides for many volumes from the same parent use case. For details, see Volume overrides.

    Presently, overrides are manually added by StorPool support only in places that require them. At a later point they will be re-created periodically, so that new volume objects are included in this analysis regularly.

  • Increases the listen backlog for the storpool_mgmt and storpool_bridge services.

19.1 revision 19.01.1468.90a9873

Released: 16 Oct 2020

  • Adds support for Micron 9300 NVMe.

  • Adds workaround for firmware issues with bnxt_en NICs (detect and reset stuck VF).

  • Improves handling of LACP mode i40e based NICs.

  • Fixes an issue with handling old-style iSCSI reservations used by VMWare 4.x and 5.x.

  • Fixes aborts in storpool_iscsi with bnxt_en driver when a large ICMP packet is received.

  • Improves StorPool API operation with large number of volumes and snapshots with multicluster environment.

  • Adds more granular flags for snapshot dematerialization status.

  • Adds fixes for iSCSI sessions list (known as empty target/initiator in CLI).

  • Adds fix for CLI not showing unnamed volumes name in attachments list.

  • Removes deprecated options from storpool.conf.example.

  • Includes updates and fixes in internal tools.

19.1 revision 19.01.1413.b807f92

Released: 26 Sep 2020

  • Updated dependencies when MLNX OFED is used.

  • Balancer fix for cases where overrides are enabled.

  • Adds bridge multicluster stability fixes.

  • Updates python bindings with missing functionality available in the present API.

  • Adds diagnostic features for the /usr/lib/storpool/monitor utility.

  • Adds handling for misbehaving i40e based NICs that could lead to aborted services.

  • Adds updates and fixes in internal tools.

19.1 revision 19.01.1376.ce07826

Released: 21 Aug 2020

  • Adds volumecare support for unnamed volumes.

  • The in-server disk tester now sends FUA writes during tests (except for –no-fua y disks).

  • Improved performance when relocating data from SSD or NVMe disks

  • Adds handling in api-ip for some obscure kernel 4.18 behavior leaving leftover routes.

  • Adds a fix for API to not show deleted snapshots addressed by global ID.

  • Adds a fix for volumeMoveRemote API call now properly handling IOPS and bandwidth limits.

  • Includes updates and fixes in internal tools.

19.1 revision 19.01.1357.39c014c

Released: 12 Aug 2020

  • Fixes a bug in handling old-style iSCSI reservations used by VMWare 4.x and 5.x

  • Adds storpool_volumecare as an installable module (More on storpool_volumecare here)

  • Adds support for Micron 7300 Pro NVMe

  • Updates and improvements in internal tools

19.1 revision 19.01.1346.dd68fa2c6

Released: 04 Aug 2020

  • Cluster volume/snapshot limit bumped to 60k volumes/snapshots.

  • StorPool API now automatically fails over in case its address vanishes for any reason.

  • The storpool_cg tool now properly handles disks with custom objects and/or entries counts.

  • Now automatically patching /etc/kdump.conf to blacklist non-important but memory intensive drivers.

  • New default TCP sysctl settings for better throughput performance for bridge enabled nodes.

  • Updated installation helper scripts handling better latest rpm/deb packages and other improvements.

19.1 revision 19.01.1293.cfcb869

Released: 11 Jul 2020

  • Improvements in storpool_initdisk tool - exit code 0 on --nvme-smart, added --json output

  • Adds spopenstack in the main source tree

  • Initial CentOS 8 build support

  • Major installation package system changes - new implementation of most core installation tools, all service files are now handled as rpm/deb packages

  • Critical fix for all systems running with hard disk drives, resiliency on power outages.

19.1 revision 19.01.1217.1635af7

Released: 02 Jul 2020

  • Improves handling of lost packets in iSCSI connections

  • Adds full cluster maintenance mode monitoring improvements

  • Server now automatically tests and returns disks on transient failures. This improves the way storage media and/or controller failures are handled.

  • Improvement in multicluster - volume create/update flags now mandatory for MultiCluster API endpoint, adds VolumeRevert operation

  • Improvement in multicluster - volume exports and remote volumes can now be listed

  • Performance improvements - more efficient bridge and API interoperability

  • Performance improvements in dematerialization

  • Adds CLI improvements - iSCSI list interfaces

  • Updates cgtool defaults - the system.slice is now with 4G limit by default

  • Adds minor fixes and improvements in balancer, build and automation tools

19.1 revision 19.01.1108.02703b8c5

Released: 04 Jun 2020

  • storpool_bridge performance improvements

  • Oracle Linux support

  • Kernel 5.4 support

  • Increase limit of total volumes and snapshots to 45000

  • Better support for LACP hashing handling

19.1 revision 19.01.1025.0baac06a6

Released: 04 Jun 2020

  • Added Maintenance mode

  • Support for volume creation with global IDs instead of volume names.

    StorPool now can allow the creation of a volume without a name provided, by assigning it a globally unique ID, by which it could be then referred.


    Globally-unique in this regard means world-wide global uniqueness, i.e. no two StorPool deployments can generate the same ID.

    This feature was developed to handle some race conditions and retry-type cases in orchestration systems in multi-cluster environment, which could lead to duplication and reuse of another volume. By allowing the storage system to ensure the uniqueness of volume identifiers, all these cases are solved.

  • Revert to snapshot functionality

  • NIC initialization and configuration is now non-blocking.

    Non-blocking NIC initialization and configuration allows for StorPool services to continue operating normally during hardware reconfigurations or errors. For example: link flaps, changing MTU or other NIC properties, adding/removing VFs to enable/disable hardware acceleration, and so on.

    This was developed to handle delays from hardware-related reconfiguration or initialization issues, as most NICs require resets, recreation of filters, or other tasks which take time for the NIC to process, and if just busy-waited would not allow the services to process other requests in the mean time.

    The feature works by sending requests to the NIC for specific changes and rechecking their progress periodically (between processing other tasks), or when waiting for other events to complete.

  • Availability of information for active iSCSI sessions

  • Acceleration support for virtio-net based adapters

  • Added support for global active operations list.

    StorPool has support for listing the active requests on disks and clients. With this change the functionality is expanded to be able to get all such data from other services (for example, storpool_bridge and storpool_iscsi) and to show some extra details. This is also expanded to be done with a single call for the whole cluster, greatly simplifying monitoring & debugging.

    This feature was developed to replace the latthreshold tool that showed all requests of clients and disks that were taking more than a set time to complete. The present implementation gets all required data without the need for sending a separate API call for each client/disk in the cluster.

  • storpool_initdisk now has extra validation of flags and devices

  • reuseServer implicit on a down disk in small clusters.

    This feature allows a volume to be created even if a cluster is on the minimum system requirements of three nodes, with a single disk missing from the cluster, or if one of the nodes is down at the moment.

    Before this change all attempts to create a volume in this state would have resulted in “Not enough servers or fault sets for replication 3” error for volumes with replication 3.

    With this feature enabled the volume will be created as if the volume was created with reuseServer. For details, see 12.22.6.  Reusing server on disk failure.

19.1 revision 19.01.878.7b1f83e3d

Released: 04 Jun 2020

  • Infrastructure to allow customers to access StorPool packages

  • Tools to help with installing new modules and services

  • Add bridge buffer size options (location update command), see also Bridge throughput performance

  • Fix for sending monitoring data in blackholed IPv6 networks

19.1 revision 19.01.759.024d1bd

Released: 18 Mar 2020

  • iSCSI TCP fixes and optimizations

  • Improvements in handling of I/O errors for NVMe devices

  • Compatibility improvement in Python bindings: fix the type of the allowRemoteExported field so that it’s not always sent to the StorPool API

19.1 revision 19.01.742.47c6e9c

Released: 06 Mar 2020

  • Major improvement in aggregation performance with a significantly more efficient internal trigger

  • Improvements in dematerialization with mixed server and controller versions in the same cluster

  • Various fixes and improvements in the in-iSCSI TCP implementation

  • Performance improvements in the storpool_controller service

  • Fix for thread priorities in some services that were leading to timeouts and service restarts

19.1 revision 19.01.719.c40dd8c

Released: 26 Feb 2020

  • Various improvements/fixes in multicluster, for details see Introduction to multi-cluster mode

  • Various iSCSI fixes and optimizations

  • A better handling of TRIMs over iSCSI

  • Improvements in balancer tools, handling some edge cases

  • Various other improvements in internal tools and services

19.1 revision 19.01.544.39e62dbad

Released: 13 Jan 2020

  • Adds setirqaff validation option

  • Adds a required default kernel configuration option for improved performance with known supported storage controllers

  • Improved storpool_bridge performance under constrained CPU and network bandwidth conditions

  • Adds colorless output for the storpool_tree utility with flags for volumes and snapshots

19.1 revision 19.01.496.6d2c5bf83

Released: 03 Dec 2019

  • Removes excessive debug print logging

  • Improved kdump setup tooling for Debian based systems

  • Support for larger set of switch hash functions for LACP and ECMP

  • Updates in fio testing framework scripts, closely following latest monitoring alerting

19.1 revision 19.01.385.fc63315ef

Released: 19 Nov 2019

  • Adds support for monitoring through IPv6 only hosts.

  • Adds hardware acceleration support for BCM57412 NetXtreme-E 10Gb Ethernet Controllers.

  • Adds improved stability of quorum keeping service in unstable network conditions.

  • Adds stability improvements in the quorum keeping service in unstable networking conditions.

  • Updates the detection for hardware accelerated NICs for the bnxt_en driver in the cgroup configuration helper tools.

  • Adds an API fix and improvements related to multi-cluster support with CloudStack deployments; for details about this mode, see Introduction to multi-cluster mode

  • Adds expansion support for Kubernetes integration.

19.1 revision 19.01.375.684a0da12

Released: 07 Nov 2019

  • Adds fix for the CLI when a volume is created without a template.

  • Improvements in logging, handles unexpected verbosity.

  • Improvements in statistics collection daemons, safer methods for data collection.

  • Improvements for stability and edge cases handling in API.

  • Improvements in storpool_cg for converged nodes, single-thread hosts migration fixes.

  • Improvements in monitoring collection scripts, now collecting and handling more data.

19.1 revision 19.01.355.896c5ebaf

Released: 23 Oct 2019

  • Improvements in API performance when balancer overrides are in place.

  • Improvements for iSCSI when persistent reservations (MicroSoft CSV) are being used with target failover.

  • Improvements for aggregation logic during recovery.

  • Adds change in CLI for template propagate, now stating propagate is mandatory on template changes.

  • Adds fix for removing the reuseServer flag for volumes/snapshots previously applied with reuseServer

  • Adds an automatic disabling through vf-udevhelper for the internal i40e based NICs LLDP listener.

  • Adds an update storpool_daemon with systemd, now no longer forking and other smaller improvements.

  • Adds more variables reported by the server_stat tool, visible also in the collected statistics for the cluster.

  • Adds stability fixes in the storpool_iscsi service.

  • Adds performance fixes with improved read latency for HDD pools.

  • Adds default configuration changes for internal command output timeouts when collecting periodic reports.

  • Improvements in storpool_cg for server cache autodetection during cgroup config migration, various other improvements.

19.1 revision 19.01.318.daa3c5938

Released: 07 Oct 2019

  • Adds CLI support for multicluster mode, for details see Introduction to multi-cluster mode

  • Improvements for iSCSI with LACP bonds, unreliable network, etc.

  • Adds fixes in python bindings

  • Improvements in cgtool chain

  • Improvements in kubernetes integration code

  • Improvements in API responsiveness during dematerialization

19.1 revision 19.01.301.f1b25e7

Released: 24 Sep 2019

  • Enables setting the SMP affinity of MegaRAID controllers

  • Adds support for CentOS 7.7 (kernel 3.10.0-1062.1.1.el7.x86_64)

  • Adds support for mixed routed/switched networks for iSCSI

  • Adds various storpool_iscsi improvements and fixes

  • Various other improvements in internal tools and services

19.1 revision 19.01.271.de5921845

Released: 12 Sep 2019

  • Improvements in the irq pinning tools for better CPU congestion avoidance on storage service CPU cores

  • Improvements in internal tools for stress testing drives

  • Adds support for non-jumbo frames routed networks

  • Adds support for hardware or no sleep options for bridge service and cgroup configuration tools

  • Adds support for upgrading 18.02 clusters with iSCSI to accelerated storpool_iscsi

  • Adds various storpool_iscsi improvements and fixes

  • Adds initial support for storpool_iscsi persistent reservations

19.1 revision

Released: 10 Sep 2019

  • Adds support for 4.18 line of kernels

  • Improvements in python-bindings, monitoring services, cgroup tools

  • Various other improvements in internal tools and services

19.1 revision 19.01.199.b98e9d4

Released: 29 Aug 2019

  • Adds priority based recovery for better handling of remote bridge recovery operations

  • Adds tools for easier upgrade from 18.02 with multi site setup

  • Adds common configuration for all supported hardware accelerated NIC drivers

  • Improvements in the tooling for enabling required hardware acceleration settings

19.1 revision

  • Adds support for multiple clusters in the same location

  • Read operations now can be forwarded through the active bridge connection for not fully recovered snapshots

  • Adds support for 4.20 and 5.0 kernel line

  • Adds significant protocol improvements for small block size and low IO queue depth read/write operations latency

  • Adds significant performance improvements for storpool_iscsi, now running only on adapters with support for hardware acceleration (i40e, ixgbe, mlx4_ib, mlx5_ib drivers)

  • Adds support for iSCSI on layer 3 only networks (routed networks)

  • Adds support for hardware accelerated bnxt driver based Broadcom adapters

  • Adds backwards compatibility with pre 19.01 release versions

  • Fixes AMD EPYC v.1 hardware sleep latency issues