StorPool 19.01 Release Change Log¶

More details for the changes are available at StorPool 19.01 Release Notes

19.01.2068.b4804fe04¶

Released: 7 may 2021

Adds fix for iSCSI compare-and-write operations when IOPS limits are applied.
Fix for BGP announcement withdrawal in routed iSCSI networks.

19.01.2016.c4bc17cda¶

Released: 30 apr 2021

Adds support for HGST Ultrastar SN100 and Micron 9100 Pro devices.
Adds json output to the storpool_ctl tool.

19.01.2012.bc34dd830¶

Released: 21 apr 2021

Adds support for Toshiba Cx5 NVMe line of drives.
The loader now invokes storpool_hugepages after the /opt directory is mounted on boot.
Improves some logging messages for better clarity in beacon, API, and bridge.

19.01.1991.f5ec6de23¶

Released 13 apr 2021

Handles small MTU in iSCSI checks for portal group and controller addresses.
Adds srcversion property for kernel modules under Debian.
Enables the storpool-contrib repository in CentOS by default.
Adds the storpool_ctl helper tool (details here)
New implementation of the hpssacli_helper tool that caches the controller output (workaround for HP Smart Array opening all disks in readwrite mode).

19.01.1957.1a9a9bb68¶

Released 30 march 2021

Fixes for snapshotSpace object properties in python bindings.
The dsnaprebase tool is now considering transient snapshots as deleted when they no longer save any space.

19.01.1949.1aea86986¶

Released 26 march 2021

Adds fix in API for the snapshotSpace call when spaceUsed is a negative value.

19.01.1946.0b0b05206¶

Released 20 march 2021

Adds fixes in iSCSI checks in storpool_stat
- Adds support for interfaces with lower MTU on the initiators
- Adds fix for abort when nodes are without access to the StorPool API.
- Adds caching for the last known addresses and MTU for each known initiator (more at Initiator addresses cache for storpool_stat).

19.01.1942.0464eda88¶

Released 16 march 2021

Adds CLI help readability changes and fixes some spelling errors.
Adds fix for storpool_beacon LACP networks and SP_IGNORE_RX_PORT=1 configuration.
Support for Kubernetes worker nodes to attach StorPool volumes via iSCSI
Not specifying an export now exports a target to all initiators for a portal group.

19.01.1907.2389092f3¶

Released 9 march 2021

Adds fix for a false positive check in the disk_tester utility.

19.01.1905.5a6c7e113¶

Released 8 march 2021

The update_* tools are now covering also the storpool_volumecare service.
Adds fix for wrong MAC address initialization for the storpool_beacon service after NIC reset.
Adds support for generating bridge interfaces with automation tools.
Adds fix for ejecting the last disk on a running storpool_server instance not flagged as ejected.
Includes various other improvements in internal tools and services

19.01.1879.4679753db¶

Released: 1 march 2021

Reliability improvements in the storpool_mgmt (API) and the storpool_bridge services.
API performance improvements in clusters with many volumes and snapshots (>20k).
Includes volumecare version 1.20 fixes and updates (8. VolumeCare Changelog).

19.01.1816.16fa37c0d¶

Released: 15 feb 2021

The unprivileged storpool account used for sending debug reports is now a system one.

19.01.1813.f4697d8c2¶

Released: 08 feb 2021

Adds remote bridge status, showing the status of remotely connected clusters.
Initial support for Ubuntu 20.04 LTS
Includes improvements and fixes in internal tools, and services.

19.01.1795.5b374e835¶

Released: 30 jan 2021

Adds support for the vfio-pci driver for NVMe devices (more at vfio-pci driver handling for NVMe devices).
Adds support for Western Digital Ultrastar SN200 NVMe device.
Includes volumecare version 1.19 fixes and updates (8. VolumeCare Changelog).
Includes small fixes and additions in internal services (service checks, storpool_abrtsync, etc.).
Fixes a problem with OpenStack components running both in a container and on the host system.
Includes fix in storpool_bd kernel module now properly showing statistics under CentOS 8 and Ubuntu’s 5.xx kernels

19.01.1732.02297f62b¶

Released: 15 jan 2021

Adds fixes for iscsi_checks in storpool_stat when network interfaces are missing.
In-server disk tester latency measurement fixes.
Adds fixes in tools for hugepages reservation (NVMe namespace detection)
Updates CentOS 8 related debugging tools.

19.01.1720.8c71b2ec3¶

Released: 12 jan 2021

Adds support for Western Digital Gold NVMe U.2 (WDS960G1D0D).
Additional per-host checks in the storpool_stat service now collects checksums for all StorPool services and related configuration files.
Adds fix for aggregation with large SATA SSD disks (7TB+).
Adds visible function symbols for easier function tracing (i.e., with perf top).
Adds small fixes in some externally visible helper tools (netbandwidth, disk_tester), internal tools and services.

19.01.1689.6a0aa758b¶

Released: 08 jan 2021

A major internal build system overhaul.
Adds initial support for the Intel NICs handled by the ice driver
Adds support for Toshiba XD5

19.01.1656.1d6d61d3f¶

Released: 22 Dec 2020

Multicluster backup locations are now supported in volumecare (8. VolumeCare Changelog).
New policy in volumecare (3.2.4. keep-daily-split).
Includes improvements and and fixes in internal tools, core and services.
Creating a template without replication is now forbidden.
Monitoring now also tracks connectivity between iSCSI controllers and initiators.

19.01.1628.1b627d0¶

Released: 07 Dec 2020

Now collecting additional configuration data in storpool_stat for additional per-host monitoring checks, includes lldpd as dependency.
Adds Volumecare version 1.18 that includes support for multiple sub-clusters sending to a backup location.
Includes improvements and and fixes in internal tools, core and services.
Adds compression and proxy authorization configuration support options for the monitoring collection scripts.
Adds the SP_MIRROR_DIR option for containerized workloads.
Adds support for the Micron 9300 MTFDHAL3T2TDR NVMe drive.
Adds compression support for monitoring data collection scripts.

19.01.1548.00e5a5633¶

Released: 27 Nov 2020

Adds fix for storpool_stat for handling statistics for unnamed volumes.
Adds support for Samsung PM173X NVMe drives.
Adds fix for iSCSI re-assign targets implementation to retry on transaction reset.
Improves storpool_stat send performance after longer periods of no connectivity
Adds storpool_initidks –no-notify option for easier StorPool disk attributes changes.
Adds reliability improvements in the in-server disk tester implementation
Adds an additional helper option to the iface-genconf tool (Auto-interface configuration)
Adds fix for disk not getting ejected by the server after a drive was re-balanced out.
Adds workaround for a bug in the i40e driver in CentOS6 based NICs (hardware acceleration)

19.01.1511.0b533fb

Released: 28 Oct 2020

Adds fixes for idle service VF resets with ixgbe driver based NICs.
Adds fixes for volumecare tags handling with unnamed volumes.
Adds initial support for CentOS 8.2 installations.
Adds tools for setting up volume overrides for many volumes from the same parent use case.
Increases the listen backlog for the storpool_mgmt and storpool_bridge services.

19.01.1468.90a9873¶

Released: 16 Oct 2020

Adds support for Micron 9300 NVMe.
Adds workaround for firmware issues with bnxt_en NICs (detect and reset stuck VF).
Improves handling of LACP mode i40e based NICs.
Fixes an issue with handling old-style iSCSI reservations used by VMWare 4.x and 5.x.
Fixes aborts in storpool_iscsi with bnxt_en driver when a large ICMP packet is received.
Improves StorPool API operation with large number of volumes and snapshots with multicluster environment.
Adds more granular flags for snapshot dematerialization status.
Adds fixes for iSCSI sessions list (known as empty target/initiator in CLI).
Adds fix for CLI not showing unnamed volumes name in attachments list.
Removes deprecated options from storpool.conf.example.
Includes updates and fixes in internal tools.

19.01.1413.b807f92¶

Released: 26 Sep 2020

Updated dependencies when MLNX OFED is used.
Balancer fix for cases where overrides are enabled.
Adds bridge multicluster stability fixes.
Updates python bindings with missing functionality available in the present API.
Adds diagnostic features for the /usr/lib/storpool/monitor utility.
Adds handling for misbehaving i40e based NICs that could lead to aborted services.
Adds updates and fixes in internal tools.

19.01.1376.ce07826¶

Released: 21 Aug 2020

Adds volumecare support for unnamed volumes.
The in-server disk tester now sends FUA writes during tests (except for –no-fua y disks).
Improved performance when relocating data from SSD or NVMe disks
Adds handling in api-ip for some obscure kernel 4.18 behavior leaving leftover routes.
Adds a fix for API to not show deleted snapshots addressed by global ID.
Adds a fix for volumeMoveRemote API call now properly handling IOPS and bandwidth limits.
Includes updates and fixes in internal tools.

19.01.1357.39c014c¶

Released: 12 Aug 2020

Fixes a bug in handling old-style iSCSI reservations used by VMWare 4.x and 5.x
Adds storpool_volumecare as an installable module (More on storpool_volumecare here)
Adds support for Micron 7300 Pro NVMe
Updates and improvements in internal tools

19.01.1346.dd68fa2c6¶

Released: 04 Aug 2020

Cluster volume/snapshot limit bumped to 60k volumes/snapshots.
StorPool API now automatically fails over in case its address vanishes for any reason.
The storpool_cg tool now properly handles disks with custom objects and/or entries counts.
Now automatically patching /etc/kdump.conf to blacklist non-important but memory intensive drivers.
New default TCP sysctl settings for better throughput performance for bridge enabled nodes.
Updated installation helper scripts handling better latest rpm/deb packages and other improvements.

19.01.1293.cfcb869¶

Released: 11 Jul 2020

Improvements in storpool_initdisk tool - exit code 0 on --nvme-smart, added --json output
Adds spopenstack in the main source tree
Initial CentOS 8 build support
Major installation package system changes - new implementation of most core installation tools, all service files are now handled as rpm/deb packages
Critical fix for all systems running with hard disk drives, resiliency on power outages.

19.01.1217.1635af7¶

Released: 02 Jul 2020

Improves handling of lost packets in iSCSI connections
Adds full cluster maintenance mode monitoring improvements

Server now automatically tests and returns disks on transient failures
Improvement in multicluster - volume create/update flags now mandatory for MultiCluster API endpoint, adds VolumeRevert operation
Improvement in multicluster - volume exports and remote volumes can now be listed
Performance improvements - more efficient bridge and API interoperability
Performance improvements in dematerialization
Adds CLI improvements - iSCSI list interfaces
Updates cgtool defaults - the system.slice is now with 4G limit by default
Adds minor fixes and improvements in balancer, build and automation tools

19.01.1108.02703b8c5¶

Released: 04 Jun 2020

storpool_bridge performance improvements
Oracle Linux support
Kernel 5.4 support
Increase limit of total volumes and snapshots to 45000
Better support for LACP hashing handling

19.01.1025.0baac06a6¶

Released: 04 Jun 2020

Maintenance mode
Support for volume creation with global IDs instead of volume names, see global ID volumes
Revert to snapshot functionality
NIC initialization and configuration is now non-blocking, see non-blocking NIC configuration
Availability of information for active iSCSI sessions
Acceleration support for virtio-net based adapters
Add support for a global active operations list
storpool_initdisk now has extra validation of flags and devices
reuseServer implicit on a down disk in small clusters, more on this here

19.01.878.7b1f83e3d¶

Released: 04 Jun 2020

Infrastructure to allow customers to access StorPool packages
Tools to help with installing new modules and services
Add bridge buffer size options (location update command), see also Bridge throughput performance
Fix for sending monitoring data in blackholed IPv6 networks

19.01.759.024d1bd¶

Released: 18 Mar 2020

iSCSI TCP fixes and optimizations
Improvements in handling of I/O errors for NVMe devices
Compatibility improvement in Python bindings: fix the type of the allowRemoteExported field so that it’s not always sent to the StorPool API

19.01.742.47c6e9c¶

Released: 06 Mar 2020

Major improvement in aggregation performance with a significantly more efficient internal trigger
Improvements in dematerialization with mixed server and controller versions in the same cluster
Various fixes and improvements in the in-iSCSI TCP implementation
Performance improvements in the storpool_controller service
Fix for thread priorities in some services that were leading to timeouts and service restarts

19.01.719.c40dd8c¶

Released: 26 Feb 2020

Various improvements/fixes in multicluster, more on mc here
Various iSCSI fixes and optimizations
A better handling of TRIMs over iSCSI
Improvements in balancer tools, handling some edge cases
Various other improvements in internal tools and services

19.01.544.39e62dbad¶

Released: 13 Jan 2020

Adds setirqaff validation option
Adds a required default kernel configuration option for improved performance with known supported storage controllers
Improved storpool_bridge performance under constrained CPU and network bandwidth conditions
Adds colorless output for the storpool_tree utility with flags for volumes and snapshots

19.01.496.6d2c5bf83¶

Released: 03 Dec 2019

Removes excessive debug print logging
Improved kdump setup tooling for Debian based systems
Support for larger set of switch hash functions for LACP and ECMP
Updates in fio testing framework scripts, closely following latest monitoring alerting

19.01.385.fc63315ef¶

Released: 19 Nov 2019

Adds support for monitoring through IPv6 only hosts.
Adds hardware acceleration support for BCM57412 NetXtreme-E 10Gb Ethernet Controllers.
Adds improved stability of quorum keeping service in unstable network conditions.
Adds stability improvements in the quorum keeping service in unstable networking conditions.
Updates the detection for hardware accelerated NICs for the bnxt_en driver in the cgroup configuration helper tools.
Adds an API fix and improvements related to multicluster support with CloudStack deployments, more on mc here
Adds expansion support for Kubernetes integration.

19.01.375.684a0da12¶

Released: 07 Nov 2019

Adds fix for the CLI when a volume is created without a template.
Improvements in logging, handles unexpected verbosity.
Improvements in statistics collection daemons, safer methods for data collection.
Improvements for stability and edge cases handling in API.
Improvements in storpool_cg for converged nodes, single-thread hosts migration fixes.
Improvements in monitoring collection scripts, now collecting and handling more data.

19.01.355.896c5ebaf¶

Released: 23 Oct 2019

Improvements in API performance when balancer overrides are in place.
Improvements for iSCSI when persistent reservations (MicroSoft CSV) are being used with target failover.
Improvements for aggregation logic during recovery.
Adds change in CLI for template propagate, now stating propagate is mandatory on template changes.
Adds fix for removing the reuseServer flag for volumes/snapshots previosly applied with reuseServer
Adds an automatic disabling through vf-udevhelper for the internal i40e based NICs LLDP listener.
Adds an update storpool_daemon with systemd, now no longer forking and other smaller improvements.
Adds more variables reported by the server_stat tool, visible also in the collected statistics for the cluster.
Adds stability fixes in the storpool_iscsi service.
Adds performance fixes with improved read latency for HDD pools.
Adds default configuration changes for internal command output timeouts when collecting periodic reports.
Improvements in storpool_cg for server cache autodetection during cgroup config migration, various other improvements.

19.01.318.daa3c5938¶

Released: 07 Oct 2019

Adds CLI support for multicluster mode, more on mc here
Improvements for iSCSI with LACP bonds, unrealiable network, etc.
Adds fixes in python bindings
Improvements in cgtool chain
Improvements in kubernetes integration code
Improvements in API responsiveness during dematerialization

19.01.301.f1b25e7¶

Released: 24 Sep 2019

Enables setting the SMP affinity of MegaRAID controllers
Adds support for CentOS 7.7 (kernel 3.10.0-1062.1.1.el7.x86_64)
Adds support for mixed routed/switched networks for iSCSI
Adds various storpool_iscsi improvements and fixes
Various other improvements in internal tools and services

19.01.271.de5921845¶

Released: 12 Sep 2019

Improvements in the irq pinning tools for better CPU congestion avoidance on storage service CPU cores
Improvements in internal tools for stress testing drives
Adds support for non-jumbo frames routed networks
Adds support for hardware or no sleep options for bridge service and cgroup configuration tools
Adds support for upgrading 18.02 clusters with iSCSI to accelerated storpool_iscsi
Adds various storpool_iscsi improvements and fixes
Adds initial support for storpool_iscsi persistent reservations

19.01.212.66fed3091¶

Released: 10 Sep 2019

Adds support for 4.18 line of kernels
Improvements in python-bindings, monitoring services, cgroup tools
Various other improvements in internal tools and services

19.01.199.b98e9d4¶

Released: 29 Aug 2019

Adds priority based recovery for better handling of remote bridge recovery operations
Adds tools for easier upgrade from 18.02 with multi site setup
Adds common configuration for all supported hardware accelerated NIC drivers
Improvements in the tooling for enabling required hardware acceleration settings

19.01.182.666093099¶

Adds support for multiple clusters in the same location
Read operations now can be forwarded through the active bridge connection for not fully recovered snapshots
Adds support for 4.20 and 5.0 kernel line
Adds significant protocol improvements for small block size and low IO queue depth read/write operations latency
Adds significant performance improvements for storpool_iscsi, now running only on addapters with support for hardware acceleration (i40e, ixgbe, mlx4_ib, mlx5_ib drivers)
Adds support for iSCSI on layer 3 only networks (routed networks)
Adds support for hardware accelerated bnxt driver based Broadcom adapters
Adds backwards compatibility with pre 19.01 release versions
Fixes AMD EPYC v.1 hardware sleep latency issues

18.02.1030.2e4eab8¶

Adds fix for storpool_beacon service backup nets detection with single-vlan and misconfigured network
Adds support for Samsung PM981 NVMe drives
Adds support for up to 7 server instances on a single node
Improves snapshot dematerialization without impact processing of other API calls
Adds fix for snapshot space miscalcuated gains for snapshots with very small amount of data
The CLI now properly shows all units in KiB, MiB, GiB to prevent any confusion
Adds a new service storpool_abrtsync, which will automatically send reports from aborted services (repsync deprecated and removed on the first run of storpool_abrtsync)
Handles per-host alerts in storpool_stat for processes in the root cgroup and unreachable API (for nodes configured to access it)
Consolidates most cron tasks in a single cron job handled by the storpool_periodic tool
Renames the storpool_initdisk’s’ --force option to the unabmiguous --wipe-all-data
Includes various other improvements in the storpool_server, storpool_beacon services, python bindings, cgroup setup tools and in the disk stress testing tool

18.02.953.79d8ee7¶

Fixes in automation for rebasing long volume-snapshot chains
Speed up TRIM operations for volumes exposed through iSCSI, now ~600GB/s instead of 2-12GB/s
Adds SP_PREFERRED_PORT option, enabling active-backup redundant networking
Improvements in relocator in aggregate heavy enviroments, such as HDD-only placement templates
Improvements in balancer tools when mixing drives in the same placement group

18.02.944.478cc9f¶

Add storpool_process tool for listing and reclassifying processes in their intended cgroups
Add the storpool_cg tool for creating cgroups based on predefined config or by autodetecting settings based on installed services and hardware
The monitor collector tool now detects the SSL CA list location
The multi-server-helper tool now supports configurations with selectable number of SSD only instances
Fix for the storpool_initdisk tool when the underlying devices are with a long PCI address
Improvements in storpool_stat collecting service, now gathering data for services network traffic and iostat for all drives, including system ones
Fixed inventory collection not updating the latest kernel booted

18.02.886.f6b2fcf20¶

The storpool_initdisk tool now handles NVMe journals for hard disk drives
The storpool_initdisk tool now lists empty NVMe partitions
Fix storpool_controller service might end up in storpool.slice cgroup on systemd daemon-reload
Add support for 4.19+ & OFED in the storpool_rdma kernel module
Fix templates status output when the same placement group is used more than once
Drops support for Ubuntu 14.04 and Debian 8 distributions
Various other improvements in internal tools and services

18.02.847-b09fd4bec¶

Workaround for bnx2x based NICs getting in partial reset when large-receive-offload is disabled
Improvements in TRIM performance on top of a StorPool volume
Improvements in recovery times with remote backups
Python bindings python3 support
Added proxy support for sending monitoring and statistics data
Various other improvements in internal tools and services

18.02.763.0aa70d7¶

Support for 4k native NVMe devices
Support for Intel Optane drives
Support for Micron 9200MAX NVMe
Support for AMD EPYC hardware sleep
Support for Intel X550 in hardware accelerated mode with ixgbe driver
Support for MTU 1500 with bnx2x and ixgbe based NICs
Updates in tools for automatic hardware acceleration configuration
Updated on-boot hugepages reservation mechanism
Performance and stability improvements in API service
Updated ixgbe driver based acceleration to evade issues with in-kernel driver
Updated ixgbe driver based acceleration with support for Debian 9 based nodes
Added Ubuntu 18.04 LTS support
Updates in statistics gathering - support for sending to secondary databases, cgroups memory statistics, various reliability and performance improvements
Updated dependencies
Various reliability and stability improvements in internal tools and services

18.02.458.ac2f823¶

Support for Kubernetes
Support for ARM64 architecture
Support for POWER architecture
Improvements in the installation procedure, VF helper included
hardware acceleration for NICs with bnx2x driver
Initial support for Alpine Linux
New API call to quickly return just the status of all volumes
Changes in the way data on the underlying drives is aggregated in a more efficient manner
New hardware support - Micron 9200 NVMe
Native 4k drives support
Various improvements in internal tools, monitoring and services

18.02.370.2b8c3c3¶

Add support for Samsung PM1725a NVMe
Add new tool, transfer_status to show the status of the currently running bridge transfers
Add new tool, sDump to print network stats in human-readable format
Fix for data transfer issues caused by the kernel remapping some of StorPool’s memory
Add preliminary support for Ubuntu 18.04
Work-around for kernel crashes caused by bugs in the ixgbe kernel driver when using acceleration
Add stats collection for per-CPU statistics, to be available in https://analytics.storpool.com/

18.02.334.c204ed2¶

Add additional logging for storpool_bridge service
Higher default parallelism for the storpool_bridge transfers
Lower CPU usage, higher efficiency for storpool_block during TRIM operations
Exit when the support for hardware sleep is not available and hardware sleep is requested
Added additional network statistics through the use of the statsDump tool and storpool_ping tools, added netlatency tool to quickly measure and report the network latency from one of the nodes to the other participants in the cluster
Added support for storpool_bd kernel module with 4.10+ kernels and storpool_rdma with 4.15+ kernels
storpool_server/storpool_initdisk/storpool_nvmed now support hotplug NVMe devices
Various other reliability fixes and improvements

18.02.206.06d240e¶

No longer ignore snapshots being deleted when re-basing long snapshot-volume chains
Fix a bug in handling aborted iSCSI operations in storpool_iscsi service
Fix in storpool_server to not forget to delete moved snapshot objects
Add support for centos 7.5 in the storpool_rdma kernel module
Add support for 4.14+ & OFED in the storpool_rdma kernel module
Add support for 4.12+ in the storpool_pci kernel module
Improve some of the diagnostic and monitoring tools
Make the “recovery” state of a task clearly visible in the output of storpool task list

18.02.178.9d29cd7¶

Disable check of BBU status for genreport
Fix service name and symlinks on CentOS 6 for the storpool_controller service
Fix installing dependency packages on Ubuntu 14.04
Fix the installation of the CentOS 6 cgmove init script
Remove “reload” for init scripts, as it’s not supported by the StorPool services and actually does “restart”
Add /usr/lib/storpool/get_tech_support - Collects useful information to be sent to support for initial troubleshooting in case of no access to the cluster.

18.02.164.7277e8c¶

Use port 443 (instead of 8086) in the storpool_stat service to get through firewalls more easily
Let the StorPool CLI prompt be customized using the SP_CLI_PROMPT variable
Add support for GCC 7
Strip the zero VLAN tag in the storpool_rdma module
Multiple fixes and improvements on the server and bridge services
Remove log entries for recovered objects (prevents syslog flooding)
Handle the new fio 3.1+ log format in the disk_tester tool
Improve the fio test scripts
Run the mgmt and bridge services in a separate memory cgroup - mgmt.slice
Add the SP_CPU_DMA_LATENCY variable to the storpool.conf.example file (more info at https://access.redhat.com/articles/65410 and https://www.kernel.org/doc/Documentation/power/pm_qos_interface.txt)
Add a check for the status of the BBU/Cache Vault of RAID controllers
Improve repsync service
Add a tool for filtering git logs
Add a tool for rotating snapshots on number of volumes
Add a tool for to quickly check the network latency to other nodes in the cluster (uses addresses in /etc/storpool.conf)