Network interfaces

The recommended mode of operation is with hardware acceleration enabled for supported network interfaces. Most NICs controlled by the i40e/ixgbe/ice (Intel), mlx4_core/mlx5_core (Nvidia/Mellanox), and bnxt (Broadcom) drivers do support hardware acceleration.

When enabled, the StorPool services are directly using the NIC, bypassing the Linux kernel. This reduces CPU usage and processing latency, and StorPool traffic is not affected by issues in Linux kernel (floods). Due to the Linux kernel being now bypassed, the entire network stack is implemented in the StorPool services.

Preparing Interfaces

There are two ways to configure the network interfaces for StorPool. One is automatic: providing the VLAN ID, the IP network(s), and the mode of operation, and leaving the IP address selection to the helper tooling. The other is semi-manual: providing explicit IP address configuration for each parameter on each of the nodes in the cluster.

Automatic configuration

The automatic configuration is performed with the net_helper tool provided by StorPool. On running the tools it selects addresses based on the SP_OURID of each node in the cluster. It requires VLAN (default 0, that is, untagged), an IP network for the storage, and pre-defined mode of operation for the interfaces in the OS. For a detailed overview, see Introduction to network interface helper.

The supported modes are exclusive-ifaces, active-backup-bond, bridge-active-backup-bond, mlag-bond, and bridge-mlag-bond. They are described in the sections below.

Note

All modes relate only to the way the kernel networks are configured. The storage services are always in active-active mode (unless configured differently) using directly the underlying interfaces. Any configuration of the kernel interfaces is solely for the purposes of other traffic, for example for access to the API.

Exclusive interfaces mode

The exclusive-ifaces mode offers the simplest possible configuration, with just the two main raw interfaces, each configured with a different address.

This mode is used mainly when there is no need for redundancy on the kernel network, usually for multipath iSCSI.

Note

Not recommended for the storage network if the API is configured on top. In such a situation it is recommended to use some of the bond modes.

Active backup bond modes

In the active-backup-bond and bridge-active-backup-bond modes, the underlying storage interfaces are added in an active-backup bond interface (named spbond0 by default), which uses an ARP monitor to select the active interface in the bond.

If the VLAN for the storage interfaces is tagged, an additional VLAN interface is created on top of the bond.

Here is a very simple example with untagged VLAN (that is, 0):

digraph G {
rankdir=LR;
image=svg;
compound=true;
sp0;
sp1;
spbond0;
spbond0 -> sp1;
spbond0 -> sp0;
}

In case the network is with a tagged VLAN, it will be created on top of the spbond0 interface.

Here is an example with tagged VLAN 100:

digraph G {
rankdir=LR;
image=svg;
compound=true;
sp0;
sp1;
spbond0;
"spbond0.100";
spbond0 -> sp1;
spbond0 -> sp0;
"spbond0.100" -> spbond0;
}

In the bridge-active-backup-bond mode, the final resolve interface is a slave of a bridge interface (named br-storage by default).

This is a tagged VLAN 100 on the bond:

digraph G {
rankdir=LR;
image=svg;
compound=true;
sp0;
sp1;
spbond0;
"br-storage";
"spbond0.100";
spbond0 -> sp1;
spbond0 -> sp0;
"br-storage" -> "spbond0.100" -> spbond0;
}

Lastly, here is a more complex example with four interfaces (sp0, sp1, sp2, sp3). The first two for the storage network are in bridge-active-backup-bond mode. The other two for the iSCSI network are in exclusive-ifaces mode. There are two additional networks on top of the storage resolve interface (in this example, 1100 and 1200). There is also an additional multipath network on the iSCSI interfaces with VLANs: 1301 on the first, and 1302 on the second iSCSI network interface.

Creating such a configuration with the net_helper tool should be done in the following way:

# net_helper genconfig sp0 sp1 sp2 sp3 \
    --vlan 100 \
    --sp-network 10.0.0.1/24 \
    --sp-mode bridge-active-backup-bond \
    --add-iface-net 1100,10.1.100.0/24 \
    --add-iface-net 1200,10.1.200.0/24 \
    --iscsi-mode exclusive-ifaces \
    --iscsicfg 1301,10.130.1.0/24:1302,10.130.2.0/24

digraph G {
rankdir=LR;
image=svg;
compound=true;
sp0;
sp1;
spbond0;
"spbond0.1100";
"spbond0.1200";
"br-storage";
sp2;
sp3;
"sp2.1301";
"sp3.1302";
spbond0 -> sp1;
spbond0 -> sp0;
"br-storage" -> "spbond0.100";
"spbond0.100" -> spbond0;
"spbond0.1100" -> spbond0;
"spbond0.1200" -> spbond0;
"sp2.1301" -> sp2;
"sp3.1302" -> sp3;
}

The tooling helps selecting automatically the addresses for the ARP-monitoring targets, if they are not overridden for better active network selection. These addresses are usually the other storage nodes in the cluster. For the iSCSI in this mode, it is best to provide explicit ARP monitoring addresses.

LACP modes

The mlag-bond and bridge-mlag-bond modes are very close to the active-backup-bond and bridge-active-backup-bond modes described above, with the notable difference that they are LACP both when specified for the main storage or the iSCSI network interfaces.

With these bond types, no additional ARP-monitoring addresses are required or being auto-generated by the tooling.

A quirk with these modes is that multipath networks for the iSCSI are being created on top of the bond interface, because there is no way to send traffic through a specific interface under the bond. Use the exclusive-iface mode for such cases.

Creating the configuration

You can create a configuration and save it as a file using the net_helper tool with the genconfig option. For mode information, see the examples provided below.

Applying the configuration

To actually apply the configuration stored in a file, use the applyifcfg option:

[root@s11 ~]# net_helper applyifcfg --from-config /etc/storpool/autonets.conf
...

Additional sub-commands available are:

up: Execute ifup/nmcli connection up on all created interfaces.
down: Execute ifdown/nmcli connection down on all created interfaces.
check: Check whether there is a configuration, or if there is a difference between the present one and a newly created one.
cleanup: Delete all network interfaces created by the net_helper tool. Useful when re-creating the same raw interfaces with a different mode.

Simple example

Here is a minimal example with the following parameters:

Interface names: sp0 and sp1 (the order is important)
VLAN ID: 42
IP Network: 10.4.2.0/24
Predefined mode mode of operation: active-backup bond on top of the storage interfaces

The example below is for a node with SP_OURID=11. Running net_helper genconfig this way will just print an example configuration:

[root@s11 ~]# storpool_showconf SP_OURID
SP_OURID=11
[root@s11 ~]# net_helper genconfig sp0 sp1 --vlan 42 --sp-network 10.4.2.0/24 --sp-mode active-backup-bond
interfaces=sp0 sp1
addresses=10.4.2.11
sp_mode=active-backup-bond
vlan=42
add_iface=
sp_mtu=9000
iscsi_mtu=9000
iscsi_add_iface=
arp_ip_targets=10.4.2.12,10.4.2.13,10.4.2.14,10.4.2.15
config_path=/etc/storpool.conf.d/net_helper.conf

To store the configuration on the file system of the node:

[root@s11 ~]# net_helper genconfig sp0 sp1 --vlan 42 --sp-network 10.4.2.0/24 --sp-mode active-backup-bond > /etc/storpool/autonets.conf

With this configuration, the net_helper applyifcfg command can be used to produce configuration for the network based on the operating system. This example is for CentOS 7 (--noop just prints what will be done):

[root@s11 ~]# net_helper applyifcfg --from-config /etc/storpool/autonets.conf --noop
Same resolve interface spbond0.42 for both nets, assuming bond
An active-backup bond interface detected
Will patch /etc/storpool.conf.d/net_helper.conf with:
________________
SP_IFACE1_CFG=1:spbond0.42:sp0:42:10.4.2.11:b:s:P
SP_IFACE2_CFG=1:spbond0.42:sp1:42:10.4.2.11:b:s:P
SP_ALL_IFACES=dummy0 sp0 sp1 spbond0 spbond0.42
________________

Executing command: iface-genconf --auto --overwrite --sp-mtu 9000 --iscsi-mtu 9000 --arp-ip-targets 10.4.2.12,10.4.2.13,10.4.2.14,10.4.2.15 --noop
Using /usr/lib/storpool, instead of the default /usr/lib/storpool
Same resolve interface spbond0.42 for both nets, assuming bond
An active-backup bond interface detected
# Noop selected, use the following commands to create the configuration file manually
cat <<_EOF_ > /etc/sysconfig/network-scripts/ifcfg-spbond0.42
# Autogenerated by /usr/sbin/iface-genconf on 2022-07-12 07:02:27

DEVICE=spbond0.42
ONBOOT=yes
TYPE=Vlan
BOOTPROTO=none
MTU=9000
VLAN=yes
PHYSDEV=spbond0
IPADDR=10.4.2.11
PREFIX=24
NM_CONTROLLED=no
_EOF_
# Noop selected, use the following commands to create the configuration file manually
cat <<_EOF_ > /etc/sysconfig/network-scripts/ifcfg-dummy0
# Autogenerated by /usr/sbin/iface-genconf on 2022-07-12 07:02:27

DEVICE=dummy0
ONBOOT=yes
TYPE=dummy
MASTER=spbond0
SLAVE='yes'
BOOTPROTO=none
MTU=9000
NM_CONTROLLED=no
_EOF_
# Noop selected, use the following commands to create the configuration file manually
cat <<_EOF_ > /etc/sysconfig/network-scripts/ifcfg-spbond0
# Autogenerated by /usr/sbin/iface-genconf on 2022-07-12 07:02:27

DEVICE=spbond0
ONBOOT=yes
TYPE=Bond
BONDING_MASTER=yes
BONDING_OPTS="mode=active-backup arp_interval=500 arp_validate=active arp_all_targets=any arp_ip_target=10.4.2.12,10.4.2.13,10.4.2.14,10.4.2.15"
BOOTPROTO=none
MTU=9000
NM_CONTROLLED=no
_EOF_
# Noop selected, use the following commands to create the configuration file manually
cat <<_EOF_ > /etc/sysconfig/network-scripts/ifcfg-sp0
# Autogenerated by /usr/sbin/iface-genconf on 2022-07-12 07:02:27

DEVICE=sp0
ONBOOT=yes
TYPE=Ethernet
MASTER=spbond0
SLAVE='yes'
BOOTPROTO=none
MTU=9000
NM_CONTROLLED=no
_EOF_
# Noop selected, use the following commands to create the configuration file manually
cat <<_EOF_ > /etc/sysconfig/network-scripts/ifcfg-sp1
# Autogenerated by /usr/sbin/iface-genconf on 2022-07-12 07:02:27

DEVICE=sp1
ONBOOT=yes
TYPE=Ethernet
MASTER=spbond0
SLAVE='yes'
BOOTPROTO=none
MTU=9000
NM_CONTROLLED=no
_EOF_

There are many additional options available, for example the name of the bond could be customized, an additional set of VLAN interfaces could be created on top of the bond interface, and so on.

Advanced example

Here is a more advanced example:

Interface names: sp0 and sp1 for StorPool, and lacp0 and lacp1 for the iSCSI service
VLAN ID: 42 for the storage interfaces
IP Network: 10.4.2.0/24
Additional VLAN ID: 43 on the bond over the storage interfaces
Storage interfaces kernel mode of operation: with a bridge with MLAG bond on top of the storage interfaces
iSCSI dedicated interfaces kernel mode of operation: with an MLAG bond
VLAN 100, and IP network 172.16.100.0/24 for a portal group in iSCSI

To prepare the configuration:

[root@s11 ~]e net_helper genconfig \
        lacp0 lacp1 sp0 sp1
        --vlan 42 \
        --sp-network 10.4.2.0/24 \
        --sp-mode bridge-mlag-bond \
        --iscsi-mode mlag-bond \
        --add-iface 43,10.4.3.0/24 \
        --iscsicfg-net 100,172.16.100.0/24 | tee /etc/storpool/autonets.conf
interfaces=lacp0 lacp1 sp0 sp1
addresses=10.4.2.11
sp_mode=bridge-mlag-bond
vlan=42
iscsi_mode=mlag-bond
add_iface=43,10.4.3.11/24
sp_mtu=9000
iscsi_mtu=9000
iscsi_add_iface=100,172.16.100.11/24
iscsi_arp_ip_targets=
config_path=/etc/storpool.conf.d/net_helper.conf

Example output:

[root@s11 ~]# net_helper applyifcfg --from-config /etc/storpool/autonets.conf --noop
Same resolve interface br-storage for both nets, assuming bond
An 802.3ad bond interface detected
Will patch /etc/storpool.conf.d/net_helper.conf with:
________________
SP_RESOLVE_IFACE_IS_BRIDGE=1
SP_BOND_IFACE_NAME=spbond0.42
SP_IFACE1_CFG=1:br-storage:lacp0:42:10.4.2.11:b:s:v
SP_IFACE2_CFG=1:br-storage:lacp1:42:10.4.2.11:b:s:v
SP_ISCSI_IFACE=sp0,spbond1:sp1,spbond1:[lacp]
SP_ALL_IFACES=br-storage dummy0 dummy1 lacp0 lacp1 sp0 sp1 spbond0 spbond0.42 spbond0.43 spbond1 spbond1.100
________________

Executing command: iface-genconf --auto --overwrite --sp-mtu 9000 --iscsi-mtu 9000 --add-iface 43,10.4.3.11/24 --iscsicfg-explicit 100,172.16.100.11/24 --noop
Using /usr/lib/storpool, instead of the default /usr/lib/storpool
Same resolve interface br-storage for both nets, assuming bond
An 802.3ad bond interface detected
# Noop selected, use the following commands to create the configuration file manually
cat <<_EOF_ > /etc/sysconfig/network-scripts/ifcfg-br-storage
# Autogenerated by /usr/sbin/iface-genconf on 2022-07-12 07:33:39

DEVICE=br-storage
ONBOOT=yes
TYPE=Bridge
BOOTPROTO=none
MTU=9000
IPADDR=10.4.2.11
PREFIX=24
NM_CONTROLLED=no
_EOF_
# Noop selected, use the following commands to create the configuration file manually
cat <<_EOF_ > /etc/sysconfig/network-scripts/ifcfg-spbond0.42
# Autogenerated by /usr/sbin/iface-genconf on 2022-07-12 07:33:39

DEVICE=spbond0.42
ONBOOT=yes
TYPE=Vlan
BRIDGE=br-storage
BOOTPROTO=none
MTU=9000
VLAN=yes
PHYSDEV=spbond0
NM_CONTROLLED=no
_EOF_
# Noop selected, use the following commands to create the configuration file manually
cat <<_EOF_ > /etc/sysconfig/network-scripts/ifcfg-dummy0
# Autogenerated by /usr/sbin/iface-genconf on 2022-07-12 07:33:39

DEVICE=dummy0
ONBOOT=yes
TYPE=dummy
MASTER=spbond0
SLAVE='yes'
BOOTPROTO=none
MTU=9000
NM_CONTROLLED=no
_EOF_
# Noop selected, use the following commands to create the configuration file manually
cat <<_EOF_ > /etc/sysconfig/network-scripts/ifcfg-spbond0
# Autogenerated by /usr/sbin/iface-genconf on 2022-07-12 07:33:39

DEVICE=spbond0
ONBOOT=yes
TYPE=Bond
BRIDGE=br-storage
BONDING_MASTER=yes
BONDING_OPTS="mode=802.3ad miimon=100 lacp_rate=1"
BOOTPROTO=none
MTU=9000
NM_CONTROLLED=no
_EOF_
# Noop selected, use the following commands to create the configuration file manually
cat <<_EOF_ > /etc/sysconfig/network-scripts/ifcfg-lacp0
# Autogenerated by /usr/sbin/iface-genconf on 2022-07-12 07:33:39

DEVICE=lacp0
ONBOOT=yes
TYPE=Ethernet
MASTER=spbond0
SLAVE='yes'
BOOTPROTO=none
MTU=9000
NM_CONTROLLED=no
_EOF_
# Noop selected, use the following commands to create the configuration file manually
cat <<_EOF_ > /etc/sysconfig/network-scripts/ifcfg-lacp1
# Autogenerated by /usr/sbin/iface-genconf on 2022-07-12 07:33:39

DEVICE=lacp1
ONBOOT=yes
TYPE=Ethernet
MASTER=spbond0
SLAVE='yes'
BOOTPROTO=none
MTU=9000
NM_CONTROLLED=no
_EOF_
# Noop selected, use the following commands to create the configuration file manually
cat <<_EOF_ > /etc/sysconfig/network-scripts/ifcfg-spbond0.43
# Autogenerated by /usr/sbin/iface-genconf on 2022-07-12 07:33:39

DEVICE=spbond0.43
ONBOOT=yes
TYPE=Vlan
BOOTPROTO=none
MTU=9000
VLAN=yes
PHYSDEV=spbond0
IPADDR=10.4.3.11
PREFIX=24
NM_CONTROLLED=no
_EOF_
# Noop selected, use the following commands to create the configuration file manually
cat <<_EOF_ > /etc/sysconfig/network-scripts/ifcfg-sp0
# Autogenerated by /usr/sbin/iface-genconf on 2022-07-12 07:33:39

DEVICE=sp0
ONBOOT=yes
TYPE=Ethernet
MASTER=spbond1
SLAVE='yes'
BOOTPROTO=none
MTU=9000
NM_CONTROLLED=no
_EOF_
# Noop selected, use the following commands to create the configuration file manually
cat <<_EOF_ > /etc/sysconfig/network-scripts/ifcfg-sp1
# Autogenerated by /usr/sbin/iface-genconf on 2022-07-12 07:33:39

DEVICE=sp1
ONBOOT=yes
TYPE=Ethernet
MASTER=spbond1
SLAVE='yes'
BOOTPROTO=none
MTU=9000
NM_CONTROLLED=no
_EOF_
# Noop selected, use the following commands to create the configuration file manually
cat <<_EOF_ > /etc/sysconfig/network-scripts/ifcfg-dummy1
# Autogenerated by /usr/sbin/iface-genconf on 2022-07-12 07:33:39

DEVICE=dummy1
ONBOOT=yes
TYPE=dummy
MASTER=spbond1
SLAVE='yes'
BOOTPROTO=none
MTU=9000
NM_CONTROLLED=no
_EOF_
# Noop selected, use the following commands to create the configuration file manually
cat <<_EOF_ > /etc/sysconfig/network-scripts/ifcfg-spbond1.100
# Autogenerated by /usr/sbin/iface-genconf on 2022-07-12 07:33:39

DEVICE=spbond1.100
ONBOOT=yes
TYPE=Vlan
BOOTPROTO=none
MTU=9000
VLAN=yes
PHYSDEV=spbond1
IPADDR=172.16.100.11
PREFIX=24
NM_CONTROLLED=no
_EOF_
# Noop selected, use the following commands to create the configuration file manually
cat <<_EOF_ > /etc/sysconfig/network-scripts/ifcfg-spbond1
# Autogenerated by /usr/sbin/iface-genconf on 2022-07-12 07:33:39

DEVICE=spbond1
ONBOOT=yes
TYPE=Bond
BONDING_MASTER=yes
BONDING_OPTS="mode=802.3ad miimon=100 lacp_rate=1"
BOOTPROTO=none
MTU=9000
NM_CONTROLLED=no
_EOF_

iSCSI configuration

The iface-genconf tool accepts an additional --iscsicfg option in the following form:

VLAN,NET0_IP/NET0_PREFIX: for single network portal groups
VLAN_NET0,NET0_IP/NET0_PREFIX:VLAN_NET1,NET1_IP/NET1_PREFIX: for multipath portal groups

The additional option is available only when also using the --auto option. This way the usage is assumed as complementary to the interface configurations initially created with the same tool, and is required to auto-detect configurations where the interfaces for the storage system and the iSCSI are overlapping.

An example of adding an additional single portal group with VLAN 100 and portal group address 10.1.100.251/24 would look like this:

iface-genconf -a --noop --iscsicfg 100,10.1.100.251/24

The above will auto-detect the operating system, the type of interface configuration used for the storage system and iSCSI, and depending on the configuration type (for example, exclusive interfaces or a bond) will print interface configuration on the console. Without the -noop option non-existing interface configurations will be created and ones that already exist will not be automatically replaced (unless iface-genconf is instructed to).

The IP addresses for each of the nodes are derived by SP_OURID, and could be adjusted with the --iscsi-ip-offset option that will be summed to the SP_OURID when constructing the IP address.

The most common case for single network portal group configuration is either with an active-backup or LACP bond configured on top of the interfaces configured as SP_ISCSI_IFACE.

For example, with SP_ISCSI_IFACE=ens2,bond0;ens3,bond0 the additional interface will be bond0.100 with IP of 10.1.100.1 for the node with SP_OURID=1, etc.

The same example for a multipath portal group with VLAN 201 for the first network and 202 for the second:

iface-genconf -a --noop --iscsicfg 201,10.2.1.251/24:202,10.2.2.251/24

In case of exclusive interfaces (ex. SP_ISCSI_IFACE=ens2:ens3), or in case of an active-backup bond configuration (for example, SP_ISCSI_IFACE=ens2,bond0:ens3,bond0) the interfaces will be configured on top of each of the underlying interfaces accordingly:

ens2.201 with IP 10.2.1.1
ens2.202 with IP 10.2.2.1

The example is assumed for a controller node with SP_OURID=1.

In case of an LACP bond (for example, SP_ISCSI_IFACE=ens2,bond0:ens3,bond0:[lacp]), all VLAN interfaces will be configured on top of the bond interface (for example bond0.201 and bond0.202 with the same addresses), but such peculiar configurations should be rare.

The --iscsicfg option could be provided multiple times for multiple portal group configurations.

You can find information about all configuration available options using iface-genconf --help. For further details and some examples, see the StorPool Ansible playbook

This feature was initially added with 19.1 revision 19.01.1548.00e5a5633 release.

Manual configuration

The net_helper tool is merely a glue-like tool covering the following manual steps:

Construct the SP_IFACE1_CFG/SP_IFACE2_CFG/SP_ISCSI_IFACE and other configuration statements based on the provided parameters (for the first and second network interfaces for the storage/iSCSI)
Execute iface_genconf that recognizes these configurations, and dumps configuration in /etc/sysconfig (CentOS 7) or /etc/network/interfaces (Debian) or using nmcli to configure with Network Manager (Alma8/Rocky8/RHEL8)
Execute /usr/lib/storpool/vf-genconf to prepare or re-create the configuration for virtual function interfaces.

Network and storage controllers interrupts affinity

The setirqaff utility is started by cron every minute. It checks the CPU affinity settings of several classes of IRQs (network interfaces, HBA, RAID), and updates them if needed. The policy is built in the script and does not require any external configuration files, apart from properly configured storpool.conf (see Node configuration options) for the present node.