--- title: "ReadWriteMany Volumes in SKS" description: "Provide ReadWriteMany volumes in SKS with a single NFS-Ganesha server on Exoscale Block Storage, with automatic recovery when the server's node fails." url: https://community.exoscale.com/product/compute/containers/recipes/nfs-rwx/ product: "Containers (SKS)" section: product last_updated: 2026-11-03 --- > For AI agents: the documentation index is at https://community.exoscale.com/llms.txt. Every page is available as markdown at `index.md` or with `Accept: text/markdown`. # ReadWriteMany Volumes in SKS This guide is provided for informational purposes. Because it involves third party tooling we cannot guarantee things still work exactly as described here. Exoscale Block Storage volumes attach to one node at a time, so the `exoscale-sbs` storage class offers `ReadWriteOnce` volumes only. When several pods on different nodes need the same files, for example shared application data or a [DuckDB](https://duckdb.org/) data directory, you need `ReadWriteMany` (RWX) volumes. The [nfs-rwx sample](https://github.com/exoscale-labs/sks-sample-manifests/tree/main/nfs-rwx) provides them with one NFS server pod that stores its data on a Block Storage volume and hands out shared volumes through a storage class named `nfs`. Everything runs inside your cluster as ordinary pods; nothing has to be installed on the nodes. > [!NOTE] > The nfs-rwx sample is not covered by the SKS SLA. It was validated on live SKS clusters, but once you deploy it, you own it. See the [repository README](https://github.com/exoscale-labs/sks-sample-manifests#what-to-expect-from-this-repository) for details. If your data does not need a POSIX filesystem, or is mostly read, consider [mounting SOS buckets](https://community.exoscale.com/product/compute/containers/how-to/sks-mountpoint-s3/index.md) instead: object storage has no single server to fail over. Use NFS when applications need regular file semantics, such as editing files in place, renaming directories or file locks. ## Prerequisites - An SKS cluster with the [Exoscale CSI add-on](https://community.exoscale.com/product/storage/block-storage/how-to/per-vol/index.md) enabled, so that the `exoscale-sbs` storage class exists - `kubectl` configured to access your cluster ## Install the NFS Server 1. Apply the manifest ```bash kubectl apply -f https://raw.githubusercontent.com/exoscale-labs/sks-sample-manifests/nfs-rwx-v0.1.2/nfs-rwx/install.yaml ``` This creates the `nfs` namespace, the NFS server with a 100 GiB Block Storage volume, the `nfs` storage class, and the node fencer described in [Node failures](#node-failures). All RWX volumes share this one Block Storage volume. To start with a larger one, change the size at install time: ```bash curl -sL https://raw.githubusercontent.com/exoscale-labs/sks-sample-manifests/nfs-rwx-v0.1.2/nfs-rwx/install.yaml \ | sed 's/storage: 100Gi/storage: 500Gi/' | kubectl apply -f - ``` 2. Verify that it is running ```bash kubectl -n nfs get pods ``` After about a minute you should see three pods in the `Running` state: ``` NAME READY STATUS RESTARTS AGE nfs-nfs-server-provisioner-0 1/1 Running 0 41s node-fencer-6b56c69b79-8xrnd 1/1 Running 0 41s node-fencer-6b56c69b79-kk6qt 1/1 Running 0 41s ``` ## Create a ReadWriteMany Volume 1. Create a file named `shared-pvc.yaml` ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: shared-data spec: accessModes: - ReadWriteMany storageClassName: nfs resources: requests: storage: 10Gi ``` 2. Apply the manifest ```bash kubectl apply -f shared-pvc.yaml ``` Volumes are provisioned dynamically, so the claim is bound right away: ```bash kubectl get pvc shared-data ``` ``` NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE shared-data Bound pvc-0c3e7c1b-4f52-4c8e-9d0e-2f6b8a1d7e55 10Gi RWX nfs 5s ``` ## Use the Volume from Several Pods 1. Create a file named `shared-app.yaml`. It runs three pods, spread over the nodes, that each write a file and list the files of the others: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: shared-app spec: replicas: 3 selector: matchLabels: app: shared-app template: metadata: labels: app: shared-app spec: topologySpreadConstraints: - maxSkew: 1 topologyKey: kubernetes.io/hostname whenUnsatisfiable: ScheduleAnyway labelSelector: matchLabels: app: shared-app containers: - name: app image: busybox:1.37 command: ["sh", "-c", "while true; do date >> /data/$HOSTNAME.log; ls /data; sleep 10; done"] volumeMounts: - name: data mountPath: /data volumes: - name: data persistentVolumeClaim: claimName: shared-data ``` 2. Apply the manifest and check the output ```bash kubectl apply -f shared-app.yaml kubectl logs deploy/shared-app --tail=3 ``` Within a minute, every pod lists the files of all three pods. NFS clients cache directory listings for up to a minute; a file opened by name is always current. ## What to Expect This is a **single NFS server, not a highly available one**. When the server restarts or moves, reads and writes on the volumes pause and then continue by themselves. Applications get no errors and file locks are kept, because volumes are mounted `hard` and the server lets clients reclaim their locks after every start (the NFSv4 grace period of 90 seconds). Measured on a 3-node SKS cluster: | Event | Pause for clients | | --- | --- | | NFS server pod restarts | About 100 seconds | | Node drain, for example during a nodepool upgrade | About 2 minutes, while the volume moves to another node | | The node running the NFS server fails | About 4 to 5 minutes, with no action needed | | Growing the Block Storage volume | About 2 minutes, see [Grow the Volume](#grow-the-volume) | It fits workloads that can tolerate a few minutes of paused I/O when something happens to the server's node. ## Node Failures Kubernetes does not restart a StatefulSet pod on another node while its node is unreachable, because the old pod might still be running. The sample therefore includes a node fencer that applies the [`node.kubernetes.io/out-of-service`](https://kubernetes.io/docs/concepts/cluster-administration/node-shutdown/#non-graceful-node-shutdown) taint to a node when all of the following hold for 60 seconds: - the Exoscale cloud controller manager reports the node's instance as stopping or stopped, - the node is `NotReady`, - the node's kubelet has stopped renewing its lease. Kubernetes then restarts the NFS server on another node. When the node is `Ready` again, the fencer removes the taint. A node whose instance is still running is never fenced. > [!IMPORTANT] > The taint applies to the whole node, so every stateful pod on it is rescheduled, not only the NFS server. See the [README](https://github.com/exoscale-labs/sks-sample-manifests/tree/main/nfs-rwx#node-crash) to disable the fencer and apply the taint manually instead. ## Grow the Volume The Exoscale CSI driver resizes Block Storage volumes only while they are detached, so the NFS server has to stop for about 30 seconds. Follow the [step-by-step procedure in the README](https://github.com/exoscale-labs/sks-sample-manifests/tree/main/nfs-rwx#growing-the-volume): stop the server, request the new size, wait for the volume, start the server. Do not change the size in `install.yaml` to resize: Kubernetes does not allow that change on a StatefulSet. ## Back Up the Volumes All RWX volumes are directories on the one Block Storage volume `nfs/data-nfs-nfs-server-provisioner-0`. A [snapshot](https://community.exoscale.com/product/storage/block-storage/how-to/snapshot/index.md) of it, taken while the server runs, captures all of them at once in a crash-consistent state: ```yaml apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: nfs-data namespace: nfs spec: volumeSnapshotClassName: exoscale-snapshot source: persistentVolumeClaimName: data-nfs-nfs-server-provisioner-0 ``` For copies outside the cluster and for restoring single applications, see [Backups](https://github.com/exoscale-labs/sks-sample-manifests/tree/main/nfs-rwx#backups) in the README. ## Limitations ### One Server, Shared Capacity All volumes live on one Block Storage volume behind one server. The size requested in a claim is not enforced: any volume can fill the whole disk. Monitor the free space with `kubectl -n nfs exec nfs-nfs-server-provisioner-0 -- df -h /export` or the `kubelet_volume_stats_available_bytes` metric, and grow the volume in time. ### Security Every pod in the cluster can reach the NFS server, root is not squashed by default and traffic is not encrypted. Use it on clusters whose workloads you trust. The [README](https://github.com/exoscale-labs/sks-sample-manifests/tree/main/nfs-rwx#security) explains how to enable root squashing. ### Databases on Shared Volumes File locks work across nodes, which keeps databases such as DuckDB or SQLite from corrupting their files, but it does not make them multi-writer databases. DuckDB, for example, allows one read-write process or several read-only processes per database file. If several pods need to write, give each its own file. ## Further Reading - [nfs-rwx sample and README](https://github.com/exoscale-labs/sks-sample-manifests/tree/main/nfs-rwx) - [Why the sample uses its own NFS-Ganesha image](https://github.com/exoscale-labs/sks-sample-manifests/tree/main/nfs-rwx/image#why-not-the-upstream-image) - [Mount SOS in SKS](https://community.exoscale.com/product/compute/containers/how-to/sks-mountpoint-s3/index.md) - [Exoscale Block Storage Documentation](https://community.exoscale.com/product/storage/block-storage/index.md)