Advanced iSCSI setup

Routed setup overview

Layer-3/routed networks present some challenges to the operation of StorPool iSCSI, unlike flat layer-2 networks:

  • routes need to be resolved for destinations based on the kernel routing table, instead of ARP;

  • floating IP addresses for the portal groups need to be accessible to the whole network;

The first task is accomplished by monitoring the kernel’s routing table, the second with an integrated BGP speaker in storpool_iscsi.

Note

StorPool’s iSCSI does not support Linux’s policy-based routing, and is not affected by iptables, nftables, or any kernel filtering/networking component.

An iSCSI deployment in a layer-3 network has the following general elements:

  • nodes with storpool_iscsi in one or multiple subnets;

  • allocated IP(s) for portal group floating IP addresses;

  • local routing daemon (bird, frr)

  • access to the network’s routing protocol.

The storpool_iscsi daemon connects to a local routing daemon via BGP and announces the floating IPs from the node those are active on. The local routing daemon talks to the network via its own protocol (BGP, OSPF or something else) and passes on the updates.

Note

In a fully routed network, the local routing daemon is also responsible for announcing the IP for cluster management (managed by storpool_mgmt)

Routed setup configuration

The following needs to be added to storpool.conf:

SP_ISCSI_ROUTED=1

In routed networks, when adding the portalGroup floating IP address, you need to specify it as /32.

Note

These are example configurations and may not be the exact fit for a particular setup. Handle with care.

Note

In the examples below, the ASN of the network is 65500, StorPool has been assigned 65512, and will need to announce 192.168.42.247.

To enable the BGP speaker in storpool_iscsi, the following snippet for storpool.conf is needed (the parameters are described in the comment above it):

# ISCSI_BGP_IP:BGP_DAEMON_IP:AS_FOR_ISCSI:AS_FOR_THE_DAEMON
SP_ISCSI_BGP_CONFIG=127.0.0.2:127.0.0.1:65512:65512

And here’s a snippet from bird.conf for a BGP speaker that talks to StorPool’s iSCSI:

# variables
myas = 65512;
remoteas = 65500;
neigh = 192.168.42.1

# filter to only export our floating IP

filter spip {
    if (net = 192.168.42.247/32) then accept;
    reject;
}


# external gateway
protocol bgp sw100g1 {
        local as myas;
        neighbor neigh as remoteas;
        import all;
        export filter spip;
        direct;
        gateway direct;
        allow local as;
}

# StorPool iSCSI
protocol bgp spiscsi {
        local as myas;
        neighbor 127.0.0.1 port 2179 as myas;
        import all;
        export all;
        multihop;
        next hop keep;
        allow local as;
}

Note

For protocols different than BGP, please note that the StorPool iSCSI exports the route to the floating IP with a next-hop of the IP address configured for the portal of the node, and this information needs to be preserved when announcing the route.

Caveats with a complex iSCSI architecture

In iSCSI portal definitions, a TCP address/port pair must be unique; only a single portal within the whole cluster may be defined at a single IP address and port. Thus, if the same StorPool iSCSI service should be able to export volumes in more than one portal group, the portals should be placed either on different ports or on different IP addresses (although it is fine that these addresses will be brought up on the same network interface on the host).

Note

Even though StorPool supports reusing IPs, separate TCP ports, and so on, the general recommendation on different portal groups is to have a separate VLAN and IP range for each one. There are lots of unknowns with different ports, security issues with multiple customers in the same VLAN, and so on.

The redirecting portal on the floating address of a portal group always listens on port 3260. Similarly to the above, different portal groups must have different floating IP addresses, although they are automatically brought up on the same network interfaces as the actual portals within the groups.

Some iSCSI initiator implementations (for example, VMware vSphere) may only connect to TCP port 3260 for an iSCSI service. In a more complex setup where a StorPool service on a single host may export volumes in more than one portal group, this might mean that the different portals must reside on different IP addresses, since the port number is the same.

For technical reasons, currently a StorPool volume may only be exported by a single StorPool service (host), even though it may be exported in different portal groups. For this reason, some care should be taken in defining the portal groups so that they may have at least some StorPool services (hosts) in common.