CKA (CNCF): Local PVs, WaitForFirstConsumer, and a Released Volume

db-4 is Pending because local storage here is a finite, node-pinned inventory and it has run out.

With WaitForFirstConsumer, the binding is deliberately deferred to the scheduler: the volume binder will not bind a claim until the scheduler has picked a node, and the scheduler will only pick a node that has an Available PV whose nodeAffinity matches that node and whose capacity satisfies the claim. There were four PVs, so at most four claims can ever bind. db-0/1/2 took three; scaling to 5 created data-db-3 and data-db-4, and db-3 took the last one. There is nothing left, so the scheduler has no node it can place db-4 on, and the PVC never binds.

Diagnose in this order:

kubectl describe pvc data-db-4      # Waiting for a volume to be created / no persistent volumes available
kubectl get pv -o custom-columns=\
  NAME:.metadata.name,STATUS:.status.phase,CAP:.spec.capacity.storage,\
  CLAIM:.spec.claimRef.name,NODE:.spec.nodeAffinity

Yes, one more PV fixes it - but only if it is built correctly. All of these are mandatory for a local PV:

• capacity.storage: 100Gi (must be >= the claim; the binder will not give a smaller volume to a bigger request)
• accessModes: [ReadWriteOnce], storageClassName: local-ssd, persistentVolumeReclaimPolicy: Retain
• spec.local.path pointing at a directory that actually exists on that node - the PV is just a pointer, nothing creates the directory
• spec.nodeAffinity with kubernetes.io/hostname: worker-N - omit this and the PV is either rejected or schedulable somewhere the directory does not exist
• spec.volumeMode: Filesystem matching the claim

Then check the target node has CPU and memory free for another db pod. Because binding is WaitForFirstConsumer, the scheduler must satisfy both the storage and the compute in one pass - free storage on a full node is useless.

Two things happen, and the second one is the trap.

Immediately: the StatefulSet pod does not watch its PVCs. db-2 keeps running with the volume still mounted - the mount is a kernel-level thing, and deleting the PVC object does not unmount it. From the inside nothing appears wrong. The PVC's finalizer clears once the volume is detached, so kubectl get pvc shows it gone while db-2 happily writes to a directory nothing is tracking.

On the next pod restart: because reclaimPolicy: Retain, pv-3 moves to Released, not Available, and it keeps spec.claimRef pointing at the deleted PVC's UID. Released PVs are ignored by the binder. So the new data-db-2 PVC never binds, db-2 goes Pending with pod has unbound immediate PersistentVolumeClaims, and kubectl get pv shows pv-3 sitting there Released while the claim is Pending - a pair that looks like it should match and does not.

Recovery (destructive, understand it before running it):

kubectl scale statefulset db --replicas=2              # ensure db-2 is gone
kubectl delete pv pv-3 && kubectl apply -f pv-3.yaml   # cleanest
# or, in place, must clear the UID too, not just the name:
kubectl patch pv pv-3 --type=json \
  -p '[{"op":"remove","path":"/spec/claimRef"}]'

The risk: pv-3's data is still on worker-3, and the moment the volume returns to Available and binds to the new claim, the old database files get reused by a fresh pod. If that is what you want, fine. If not, you have just silently handed stale data to a new replica - which in a 3-member db cluster means a member with a divergent dataset. Priorities: scale the StatefulSet down before touching PVCs, back up the directory on the node first, and never delete a PVC to 'reset' stateful data without a restore plan.

Because kubernetes.io/no-provisioner volumes cannot be expanded at all. The API server rejects the patch outright:

persistentvolumeclaims "data-db-0" is forbidden:
  only dynamically provisioned pvc can be resized and
  the storageclass that provisions the pvc must support resize

The check is not about allowVolumeExpansion: true on the StorageClass - you can set that flag here and it will change nothing. Expansion requires something to actually grow the underlying block device or filesystem, and with a hand-made local PV there is no CSI driver to call; the 'provisioner' is a human. The same applies to hostPath-style PVs.

So the options are:

1. Migrate to a bigger PV. Add a new 200Gi PV pinned to a node, then create a new StatefulSet or a parallel PVC pointing at it, copy the data (rsync over a temporary pod that mounts both, or a dump/restore for the database), cut over, and keep the old PV around until you are satisfied. This is the standard and safe path for a stateful workload.
2. Move the workload onto a real CSI driver that supports allowVolumeExpansion: true. Even then: expansion is online for most filesystem CSI drivers but block volumes still need the pod restarted to pick up the new size, and shrinking is never supported - you would have to create a smaller volume and migrate.
3. Do not try to resize by editing the PV. Changing spec.capacity on a bound PV does not change the on-disk device and does not change the PVC; it just makes your inventory lie to the next person.

Also confirm db is not relying on the PVC being a fixed size anywhere - a database that was sized to fill 100Gi will now need re-tuning after the move. Diagnose the exact rejection with:

kubectl describe pvc data-db-0 | sed -n '/Conditions/,$p'
kubectl get storageclass local-ssd -o yaml | grep -i -e provisioner -e allowVolumeExpansion