ReadWriteMany Volumes in SKS

Exoscale Block Storage volumes attach to one node at a time, so the exoscale-sbs storage class offers ReadWriteOnce volumes only. When several pods on different nodes need the same files, for example shared application data or a DuckDB data directory, you need ReadWriteMany (RWX) volumes.

The nfs-rwx sample provides them with one NFS server pod that stores its data on a Block Storage volume and hands out shared volumes through a storage class named nfs. Everything runs inside your cluster as ordinary pods; nothing has to be installed on the nodes.

If your data does not need a POSIX filesystem, or is mostly read, consider mounting SOS buckets instead: object storage has no single server to fail over. Use NFS when applications need regular file semantics, such as editing files in place, renaming directories or file locks.

Prerequisites

• An SKS cluster with the Exoscale CSI add-on enabled, so that the exoscale-sbs storage class exists
• kubectl configured to access your cluster

Install the NFS Server

1. Apply the manifest

kubectl apply -f https://raw.githubusercontent.com/exoscale-labs/sks-sample-manifests/nfs-rwx-v0.1.2/nfs-rwx/install.yaml

This creates the nfs namespace, the NFS server with a 100 GiB Block Storage volume, the nfs storage class, and the node fencer described in Node failures.

All RWX volumes share this one Block Storage volume. To start with a larger one, change the size at install time:

curl -sL https://raw.githubusercontent.com/exoscale-labs/sks-sample-manifests/nfs-rwx-v0.1.2/nfs-rwx/install.yaml \
  | sed 's/storage: 100Gi/storage: 500Gi/' | kubectl apply -f -

2. Verify that it is running

kubectl -n nfs get pods

After about a minute you should see three pods in the Running state:

NAME                           READY   STATUS    RESTARTS   AGE
nfs-nfs-server-provisioner-0   1/1     Running   0          41s
node-fencer-6b56c69b79-8xrnd   1/1     Running   0          41s
node-fencer-6b56c69b79-kk6qt   1/1     Running   0          41s

Create a ReadWriteMany Volume

1. Create a file named shared-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shared-data
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: nfs
  resources:
    requests:
      storage: 10Gi

2. Apply the manifest

kubectl apply -f shared-pvc.yaml

Volumes are provisioned dynamically, so the claim is bound right away:

kubectl get pvc shared-data

NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
shared-data   Bound    pvc-0c3e7c1b-4f52-4c8e-9d0e-2f6b8a1d7e55   10Gi       RWX            nfs            <unset>                 5s

Use the Volume from Several Pods

1. Create a file named shared-app.yaml. It runs three pods, spread over the nodes, that each write a file and list the files of the others:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: shared-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: shared-app
  template:
    metadata:
      labels:
        app: shared-app
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: shared-app
      containers:
        - name: app
          image: busybox:1.37
          command: ["sh", "-c", "while true; do date >> /data/$HOSTNAME.log; ls /data; sleep 10; done"]
          volumeMounts:
            - name: data
              mountPath: /data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: shared-data

2. Apply the manifest and check the output

kubectl apply -f shared-app.yaml
kubectl logs deploy/shared-app --tail=3

Within a minute, every pod lists the files of all three pods. NFS clients cache directory listings for up to a minute; a file opened by name is always current.

What to Expect

This is a single NFS server, not a highly available one. When the server restarts or moves, reads and writes on the volumes pause and then continue by themselves. Applications get no errors and file locks are kept, because volumes are mounted hard and the server lets clients reclaim their locks after every start (the NFSv4 grace period of 90 seconds).

Measured on a 3-node SKS cluster:

It fits workloads that can tolerate a few minutes of paused I/O when something happens to the server’s node.

Node Failures

Kubernetes does not restart a StatefulSet pod on another node while its node is unreachable, because the old pod might still be running. The sample therefore includes a node fencer that applies the node.kubernetes.io/out-of-service taint to a node when all of the following hold for 60 seconds:

• the Exoscale cloud controller manager reports the node’s instance as stopping or stopped,
• the node is NotReady,
• the node’s kubelet has stopped renewing its lease.

Kubernetes then restarts the NFS server on another node. When the node is Ready again, the fencer removes the taint. A node whose instance is still running is never fenced.

Grow the Volume

The Exoscale CSI driver resizes Block Storage volumes only while they are detached, so the NFS server has to stop for about 30 seconds. Follow the step-by-step procedure in the README: stop the server, request the new size, wait for the volume, start the server.

Do not change the size in install.yaml to resize: Kubernetes does not allow that change on a StatefulSet.

Back Up the Volumes

All RWX volumes are directories on the one Block Storage volume nfs/data-nfs-nfs-server-provisioner-0. A snapshot of it, taken while the server runs, captures all of them at once in a crash-consistent state:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: nfs-data
  namespace: nfs
spec:
  volumeSnapshotClassName: exoscale-snapshot
  source:
    persistentVolumeClaimName: data-nfs-nfs-server-provisioner-0

For copies outside the cluster and for restoring single applications, see Backups in the README.

Limitations

One Server, Shared Capacity

All volumes live on one Block Storage volume behind one server. The size requested in a claim is not enforced: any volume can fill the whole disk. Monitor the free space with kubectl -n nfs exec nfs-nfs-server-provisioner-0 -- df -h /export or the kubelet_volume_stats_available_bytes metric, and grow the volume in time.

Security

Every pod in the cluster can reach the NFS server, root is not squashed by default and traffic is not encrypted. Use it on clusters whose workloads you trust. The README explains how to enable root squashing.

Databases on Shared Volumes

File locks work across nodes, which keeps databases such as DuckDB or SQLite from corrupting their files, but it does not make them multi-writer databases. DuckDB, for example, allows one read-write process or several read-only processes per database file. If several pods need to write, give each its own file.

Further Reading

• nfs-rwx sample and README
• Why the sample uses its own NFS-Ganesha image
• Mount SOS in SKS
• Exoscale Block Storage Documentation