Persistent storage on Kubernetes is one of those features that looks straightforward in the tutorials and becomes tangled in practice. PersistentVolumeClaim, StorageClass, CSI driver, dynamic provisioning, volume binding modes, expansion, snapshots — there are a lot of moving parts. This post is what each one actually does and the patterns that make them work in production.
The Layers
Kubernetes storage has several layers stacked on top of each other:
- CSI Driver. The thing that actually talks to the storage system (EBS, GCE PD, NFS, Ceph, etc.).
- StorageClass. A template that defines how volumes are provisioned (which CSI driver, with what parameters).
- PersistentVolume (PV). An actual storage volume, either statically created or dynamically provisioned.
- PersistentVolumeClaim (PVC). A pod's request for storage. References a StorageClass, gets bound to a PV.
Most production setups use dynamic provisioning: a pod creates a PVC, the StorageClass provisions a PV, and the pod gets storage. The PV is created on demand.
StorageClass Configuration
A StorageClass defines what kind of storage gets provisioned.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: fast-ssd
provisioner: ebs.csi.aws.com
parameters:
type: gp3
iops: "3000"
throughput: "125"
encrypted: "true"
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
allowVolumeExpansion: true
The fields that matter:
- provisioner: the CSI driver name.
- parameters: storage-specific config (volume type, IOPS, encryption).
- volumeBindingMode: when to provision (covered below).
- reclaimPolicy: what to do when the PVC is deleted (Delete or Retain).
- allowVolumeExpansion: whether the volume can grow after provisioning.
Most production clusters define multiple StorageClasses for different workloads: fast-ssd for databases, standard for general workloads, cheap-hdd for backups.
Volume Binding Mode
The most common configuration bug. The two options:
Immediate. As soon as the PVC is created, a PV is provisioned. The PV might be in a different zone than where the pod eventually schedules.
WaitForFirstConsumer. Provisioning waits until a pod actually uses the PVC. The PV is provisioned in the same zone as the pod.
For multi-zone clusters: always use WaitForFirstConsumer. With immediate binding, a pod can find itself unable to schedule because its volume is in another zone.
volumeBindingMode: WaitForFirstConsumer
Setting this once on every StorageClass prevents an entire class of "pod is pending forever" incidents.
Reclaim Policy
When a PVC is deleted, what happens to the PV (and the underlying storage)?
- Delete: the PV and the underlying volume are deleted. Data is gone.
- Retain: the PV is kept but unbound; the underlying volume is preserved. Manual intervention to reuse.
For production databases and anything else where data loss is unacceptable: Retain. For ephemeral workloads (cache, scratch): Delete.
Most StorageClasses default to Delete. Reset this thoughtfully.
Volume Expansion
By default, a Kubernetes volume cannot grow. Setting allowVolumeExpansion: true lets you edit a PVC's spec.resources.requests.storage and have the volume expand.
kubectl patch pvc my-db-data -p '{"spec":{"resources":{"requests":{"storage":"50Gi"}}}}'
The volume grows; depending on the filesystem, the pod may need to be restarted to see the new size.
Expansion is one-way. You cannot shrink a volume through Kubernetes APIs — the workaround is provisioning a new smaller volume and copying data.
CSI Drivers
CSI — Container Storage Interface — is the spec for storage drivers. Each storage system has its own driver:
- AWS EBS —
ebs.csi.aws.com - AWS EFS —
efs.csi.aws.com - GCP Persistent Disk —
pd.csi.storage.gke.io - Azure Disk —
disk.csi.azure.com - Azure File —
file.csi.azure.com - NFS —
nfs.csi.k8s.io - Ceph (RBD or CephFS) —
rbd.csi.ceph.com,cephfs.csi.ceph.com - Longhorn (Kubernetes-native distributed storage) —
driver.longhorn.io
Managed Kubernetes services install the cloud-provider CSI drivers automatically. Self-managed clusters need to install them.
The driver determines what storage features you can use. EBS supports volume expansion and snapshots; EFS supports concurrent multi-pod access; NFS supports concurrent access but no fancy features.
Access Modes
A PV declares what access modes it supports:
- ReadWriteOnce (RWO): one node at a time can mount it. Most block storage (EBS, GCE PD).
- ReadOnlyMany (ROX): many nodes can mount it read-only.
- ReadWriteMany (RWX): many nodes can mount it read-write. NFS, EFS, CephFS, some others.
- ReadWriteOncePod: one specific pod at a time, even on the same node.
The access mode constrains pod scheduling. A Deployment of 3 replicas with an RWO PVC can only run one pod at a time. Use a StatefulSet (one PVC per pod) or switch to an RWX storage type.
Snapshots
Volume snapshots let you capture a point-in-time copy of a PV. The CSI snapshot API provides this for compatible drivers.
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: db-snapshot-2026-10-17
spec:
volumeSnapshotClassName: ebs-snapshot
source:
persistentVolumeClaimName: my-db-data
The snapshot is a CRD; the underlying storage system creates the actual snapshot. You can later create a new PVC from the snapshot, restoring the data.
Snapshots are essential for backups but are not a backup strategy. Snapshots stored in the same cloud account and region are not safe from account-level disasters. A real backup strategy involves snapshots plus offsite replication.
Storage Selection by Workload
Different workloads want different storage:
Databases (Postgres, MySQL): RWO block storage with high IOPS. EBS gp3 with provisioned IOPS for AWS. Premium SSD for Azure.
Shared file storage (uploads, exports): RWX file storage. EFS, Azure Files, NFS.
Caches: Usually ephemeral; emptyDir or local SSD. Persistent storage is overkill.
Object storage needs (logs, backups, large unstructured data): Skip Kubernetes PVs. Use S3, GCS, Azure Blob directly. PVCs for object storage are an anti-pattern.
Stateful databases that need replication: Operators (e.g., the Postgres Operator) manage PVCs across pods. The operator handles topology.
Common Pitfalls
Zone mismatches. Pod in zone A, volume in zone B. Use WaitForFirstConsumer.
Insufficient IOPS. Default gp2 EBS volumes are slow. Set IOPS explicitly for databases.
Snapshot expiration. Snapshots accumulate without lifecycle policies. Set retention rules.
Deleting PVCs accidentally. With reclaimPolicy: Delete, the data is gone. Use Retain for important data.
Multiple pods on RWO. Cannot work. Use a StatefulSet or RWX storage.
No backups. Snapshots are not backups; they live in the same cloud account. Replicate to another region or to S3.
A Production Storage Setup
For a typical Laravel + Postgres on Kubernetes:
# StorageClass for databases
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: db-storage
provisioner: ebs.csi.aws.com
parameters:
type: gp3
iops: "5000"
throughput: "250"
encrypted: "true"
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Retain
allowVolumeExpansion: true
# StatefulSet using the PVC
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: postgres
spec:
serviceName: postgres
replicas: 1
template:
spec:
containers:
- name: postgres
image: postgres:16
volumeMounts:
- name: data
mountPath: /var/lib/postgresql/data
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: [ReadWriteOnce]
storageClassName: db-storage
resources:
requests:
storage: 100Gi
Combined with a CronJob that creates daily snapshots and an external backup process that copies snapshots offsite, you have a production-grade storage setup.
What the CSI Specification Gets Wrong
CSI made storage modular but added complexity. The result is:
- Many similar-looking but subtly different drivers
- Feature support that varies per driver
- Configuration that lives in multiple CRDs and StorageClass parameters
- Debugging that requires understanding which layer the issue is in
The fix is documentation and testing. Know which CSI driver you are using, what features it supports, and what its known issues are.
When to Avoid Kubernetes Storage
For some workloads, Kubernetes storage is the wrong answer. Consider:
- Managed databases. RDS, Cloud SQL, Aurora — let the cloud manage durable storage. Worth the cost for most production databases.
- Object storage. S3, GCS, Azure Blob for any large unstructured data. PVCs for this scale are not the right tool.
- External caches. ElastiCache, Memorystore. Caches do not need Kubernetes storage.
Most Kubernetes workloads should be stateless. The ones that are stateful — databases, queues — often run better as managed services.
Designing a storage setup for a Kubernetes workload and not sure which combination of StorageClass, CSI driver, and access mode is right? We help teams pick storage that survives the workload's actual patterns. scopeforged.com