Kubernetes Integration

1. Introduction

StorPool provides and maintains a Kubernetes integration that makes it possible for the Kubernetes cluster to use transparently StorPool volumes as persistent storage volumes for its data.

2. Requirements

• Working StorPool cluster

• Working Kubernetes deployment

• All Kubernetes workers are joined to StorPool cluster as clients

• The kubernetes StorPool module installed on all Kubernetes workers as well as on all nodes with storpool_mgmt service running

• Installed kubectl on all nodes with storpool_mgmt service running

• The jq tool used for parsing JSON output from the command line

[k8s-master]# yum --enablerepo=extras install -y epel-release
[k8s-master]# yum install -y jq

3. Installation

Create a Kubernetes ServiceAccount, ClusterRole and ClusterRoleBinding for the StorPool provisioner and give it the required permissions.

Note

StorPool provisioner configuration can be found in /usr/share/doc/storpool/examples/kubernetes/storpool-provisioner.yaml

[k8s-master]# cat > storpool-provisioner.yaml <<_EOF_
kind: ServiceAccount
apiVersion: v1
metadata:
  name: storpool-provisioner
  namespace: kube-system

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: storpool-provisioner
  namespace: kube-system
rules:
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: [""]
    resources: ["namespaces"]
    verbs: ["get"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "update", "create", "delete", "patch"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["csinodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments/status"]
    verbs: ["patch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: storpool-provisioner-role
  namespace: kube-system
subjects:
  - kind: ServiceAccount
    name: storpool-provisioner
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: storpool-provisioner
  apiGroup: rbac.authorization.k8s.io

---
apiVersion: storage.k8s.io/v1beta1
kind: CSIDriver
metadata:
  name: csi-driver.storpool.com
spec:
  attachRequired: false
  podInfoOnMount: false
  volumeLifecycleModes: # added in Kubernetes 1.16
  - Persistent
  - Ephemeral
_EOF_

[k8s-master]# kubectl apply -f storpool-provisioner.yaml

Prepare a kubeconfig file

Note

In the follwing example we are borrowing the certificate-authority-data and kubernetes master’s endpoint from ~/.kube/config on Kubernetes master.

[k8s-master]# CLUSTER_NAME=cluster1
[k8s-master]# USER_NAME=storpool

[k8s-master]# SP_SECRET_NAME=$(kubectl get serviceaccounts --namespace=kube-system storpool-provisioner -o json | jq -r '.secrets[0].name')
[k8s-master]# echo "SP_SECRET_NAME=$SP_SECRET_NAME"

[k8s-master]# PROVISIONER_TOKEN=$(kubectl get secret --namespace=kube-system "$SP_SECRET_NAME" -o json | jq -r .data.token | base64 -d)
[k8s-master]# echo "PROVISIONER_TOKEN=$PROVISIONER_TOKEN"

[k8s-master]# head -n5 ~/.kube/config > kubeconfig

[k8s-master]# cat >>kubeconfig <<_EOF_
  name: $CLUSTER_NAME
contexts:
- context:
    cluster: $CLUSTER_NAME
    user: $USER_NAME@$CLUSTER_NAME
  name: $CLUSTER_NAME
current-context: ""
kind: Config
preferences: {}
users:
- name: $USER_NAME@$CLUSTER_NAME
  user:
    token: $PROVISIONER_TOKEN
_EOF_

Note

The node config file supports multiple clusters, configurations and/or users. If there are more Kubernetes clusters, they need to be added to the config file.

Example configuration file with two kubernetes clusters:

apiVersion: v1
kind: Config
current-context: ""
preferences: {}

clusters:
- cluster:
    certificate-authority-data: DQ0FTSXdEUVlKS29aSWh2Y05BUUVCQl...
    server: https://10.8.111.253:6443
  name: $CLUSTER_NAME_1

- cluster:
    certificate-authority-data: 1Q5a0VJdUtmUnd2NkhpRwpNcWJDSHNT...
    server: https://10.8.222.253:6443
  name: $CLUSTER_NAME_2

contexts:
- context:
    cluster: $CLUSTER_NAME_1
    user: $USER_NAME_1@$CLUSTER_NAME_1
  name: $CLUSTER_NAME_1

- context:
    cluster: $CLUSTER_NAME_2
    user: $USER_NAME_2@$CLUSTER_NAME_2
  name: $CLUSTER_NAME_2

users:
- name: $USER_NAME_1@$CLUSTER_NAME_1
  user:
    token: $PROVISIONER_TOKEN_FOR_CLUSTER_1

- name: $USER_NAME_2@$CLUSTER_NAME_2
  user:
    token: $PROVISIONER_TOKEN_FOR_CLUSTER_2

Copy /etc/storpool/kubeconfig on every node with the storpool_mgmt service running.

To validate the configuration, the following command should not return an error:

[sp-mgmt-node]# kubectl --kubeconfig=/etc/storpool/kubeconfig --context=cluster1 get pv

Note

Validate from every node with the storpool_mgmt service running.

The name of the cluster (cluster1 in the above example) must be registered via the StorPool CLI. Execute the following once on a node that is a member of the StorPool cluster and configured to access the API:

[k8s-master]# storpool kubernetes add name cluster1

To check Kubernetes status in StorPool

[k8s-master]# storpool kubernetes list
[k8s-master]# storpool kubernetes status

Enable and start storpool_kubcsi.service on each Kubernetes worker node

[k8-node]# systemctl enable --now storpool_kubcsi.service

4. Usage example

Create a storage class in Kubernetes. Create a new template for the kubernetes cluster, for accountability reasons.

Hint

• Uncomment the annotation if this is to be based as default class.

• Update the template if necessary, k8s-nvme-only is just an example name for the template in StorPool

[k8s-master]# TEMPLATE="k8s-nvme-only"

[k8s-master]# cat > storageClass.yaml <<_EOF_
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: storpool-csi
  namespace: kube-system
    #annotations:
      #storageclass.kubernetes.io/is-default-class: "true"
provisioner: csi-driver.storpool.com
reclaimPolicy: Delete
parameters:
  template: "$TEMPLATE"
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
_EOF_

[k8s-master]# kubectl apply -f "storageClass.yaml"

[k8s-master]# kubectl get sc

Now PersistentVolumeClaim could be created and should get auto provisioned.

[k8s-master]# cat > pvclaim.yaml <<_EOF_
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-storpool-csi-request
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: storpool-csi
  _EOF_

[k8s-master]# kubectl apply -f "pvclaim.yaml"

[k8s-master]# kubectl get pvc

The persistent claim should be listed, and its status should change to Pending, if everything is working. The newly created volume should be listed in the pv’s:

[k8s-master]# kubectl get pv

Hint

In case this is not happening, kubectl describe pvc my-storpool-csi-request, might list a StorPool generated event e.g. noResources, or template missing, more info will be available in the storpool_mgmt API logs (/var/log/debug and /var/log/storpool/mgmt.log).

Create a pod:

[k8s-master]# cat > pod.yaml <<_EOF_
kind: Pod
apiVersion: v1
metadata:
  name: task-pv-pod
spec:
  volumes:
    - name: task-pv-storage
      persistentVolumeClaim:
        claimName: my-storpool-csi-request
  containers:
    - name: task-pv-container
      image: nginx
      ports:
        - containerPort: 80
          name: "http-server"
      volumeMounts:
        - mountPath: "/usr/share/nginx/html"
          name: task-pv-storage
_EOF_

[k8s-master]# kubectl apply -f pod.yaml

[k8s-master]# kubectl get pod

The integration now supports also live expansion of the filesystem. In order to trigger a live expansion use kubectl edit pvc my-storpool-csi-request and bump the size in spec -> resources -> requests -> storage.

Note

Use kubectl exec -it task-pv-pod -- /bin/sh -c "kill 1" if you’d like to restart the container and update the size immediately.