CKA (Other exams): A Released PersistentVolume and a recreated PVC: restoring a StatefulSet volume without losing data

Why Pending. Binding is a two-sided handshake. After the PVC was deleted, the PV could not be reclaimed (policy is Retain), so it moved to Released - data intact, but explicitly not available to any claim. Its claimRef still points at the old claim by UID, and the newly created PVC has a different UID. The PersistentVolume controller will not bind a claim to a Released PV, and the new PVC cannot be dynamically provisioned either because the StorageClass named manual has no provisioner. Result: a PVC with WaitForFirstConsumer-free static matching, no candidate, Pending.

Why the pod still writes. Deleting a PVC removes an API object, not a mount. The kubelet set up the bind mount when the container started and does not re-validate the claim while the container lives, so db-0 keeps its file handle to the same directory or block device. That is exactly why the data looks fine and exactly why this is dangerous: nothing is protecting it. If db-0 restarts, the container fails to start (volume is already exclusively attached or an unbound PVC), and the volume is left in a state that is harder to reason about than it is right now.

Diagnose with kubectl get pv pv-db-0 -o yaml (read claimRef.uid), kubectl get pvc data-db-0 -o yaml (read metadata.uid), and kubectl describe pvc data-db-0 -n data for the binder events.

Stop the writer first. An unclaimed volume with an active process on it is a race. kubectl scale statefulset db -n data --replicas=0 (or --replicas=2, which removes db-0 under OrderedReady semantics and leaves the other two) - pick scale-to-zero if you can afford a few seconds of downtime, because it is unambiguous.

Then, assuming you want to keep pv-db-0's data:

# the new PVC never bound, so it holds no data and can go
kubectl -n data delete pvc data-db-0

# free the PV from the dead claim
kubectl patch pv pv-db-0 -p '{"spec":{"claimRef":null}}'

# recreate the claim with the original name/size/class
kubectl apply -f pvc-db-0.yaml
# then scale back up
kubectl scale statefulset db -n data --replicas=3

Alternatively patch claimRef to the new PVC's UID in one step (claimRef: {namespace: data, name: data-db-0, uid: <new-uid>}), which skips the delete/recreate of the PVC. Either way, verify afterwards by exec-ing into db-0 and checking a known marker - a row count or a sentinel file - not just by kubectl get pvc saying Bound.

Timing: a few minutes, dominated by pod termination and the StatefulSet's ordered restart.

Is delete-and-recreate faster? No, and it is riskier. With Retain the backing directory or disk survives the PV object, so it usually works - but the PV protection finalizer keeps the object Terminating while db-0 still holds the mount, which means you must stop the pod anyway. And you then have to reconstruct capacity, nodeAffinity, hostPath/local path and storageClassName exactly; one wrong field points the database at an empty directory and the mistake looks like success until someone reads the data.

Treat it as a permissions problem first, because it is one: a human deleted a live PVC in a stateful namespace.

• RBAC. Give operators a Role in data that allows get/list/watch on PVCs and create only via a reviewed template; reserve delete on persistentvolumeclaims for a break-glass group. kubectl auth can-i delete pvc -n data --as=operator is a one-line audit.
• Make the destructive action expensive. A short kubectl delete --dry-run=server habit plus a namespace annotation documenting what backs each PVC. For cluster-wide safety, a ValidatingAdmissionPolicy or a Kyverno/Gatekeeper rule denying DELETE on PVCs carrying a backup=true label is cheap and effective.
• Reclaim policy as a deliberate choice, not a default. Retain protected the data here but created the stuck-Released state; it is right for database volumes and wrong for scratch space. Say so per StorageClass, and document the recovery procedure (clear claimRef) where the on-call will find it, because almost nobody remembers it at 03:00.
• Backups, not just retention. Retain survives an accidental kubectl delete; it does not survive a bad rm -rf, a corrupted filesystem, or the node's disk dying. Volume snapshots or logical dumps are the only thing that covers those.
• Finally, note that StatefulSet PVCs are deliberately not owner-referenced to pods - deleting the StatefulSet does not delete the claims. That is a feature, and it means kubectl delete sts is not a cleanup shortcut.