Zone-Bound PVCs Leave Two Rack Pods Pending After a Node-Pool Refresh
Your team runs a six-node CassandraDatacenter (dc1) on EKS: three racks r1, r2, r3, two nodes per rack, NetworkTopologyStrategy RF=3 for the production keyspace. Each rack is pinned with nodeAffinity to one zone (us-east-1a/b/c), and each pod's PVC is a 2 TiB gp3 volume bound immediately. During a zonal node-pool refresh, the platform team scaled the us-east-1b node group to zero and replaced it with nodes in us-east-1c. Two hours later, both r2 pods are Pending. kubectl describe shows node(s) had volume node affinity conflict; events name PVCs whose PVs are in us-east-1b. The remaining four pods are Ready, but nodetool status shows the r2 nodes as DN. The application is still serving reads and writes at QUORUM, though latency has doubled. A product owner asks whether to delete the PVCs so the pods can reschedule in us-east-1c. No backups have run since yesterday.
1. Diagnose why the r2 pods cannot schedule, and state what data-safety checks you perform before changing anything.
Read the worked answer
The pods are pinned to us-east-1b twice: once by the CassandraDatacenter rack nodeAffinity, and once by the PV's node affinity. The PVCs were bound with Immediate volume binding, so each PV was provisioned in the zone where the pod first ran. EBS volumes are zonal; the PV can only attach to a node in us-east-1b. With the us-east-1b node group at zero, no node satisfies either constraint, so the scheduler reports a volume node affinity conflict.
Deleting the PVCs would destroy the only replica of any RF=1 data on those nodes and would force a new node identity. The old DN nodes remain in the ring and can cause token collision when replacements try to join. Before touching anything I would check nodetool status and nodetool gossipinfo to determine whether the nodes are truly dead or merely unreachable; list keyspaces and replication with DESCRIBE KEYSPACES to find RF=1 or RF greater than the rack count; confirm the last successful backup; check hinted handoff and client error rates; and verify whether the us-east-1b node pool can be restored. I would not delete PVCs until the old nodes have been removed from the ring and the data-loss exposure is understood.
2. Compare restoring zone-b capacity with replacing the r2 nodes in zone c. Give the ordered steps for the safer path and the conditions under which the replacement path is acceptable.
Read the worked answer
The safer path is to restore the us-east-1b node group with the same zone label, instance profile, and taints, then let the scheduler place the r2 pods there. Because the PVCs and PVs are still bound, the pods should attach their existing volumes and rejoin as the same nodes. Do this one pod at a time and watch nodetool status, nodetool netstats, and client latency. This preserves the one-rack-per-zone layout and avoids moving data.
If zone-b capacity cannot be restored, the replacement path requires strict ordering. First confirm the old r2 nodes are permanently gone and that no unique data exists. From a live node, remove each dead node from the ring using its host ID (nodetool removenode <host-id>), waiting for removal to complete. Only then delete the corresponding PVCs and let the operator recreate the pods. If you change rack affinity, keep one rack per zone; do not collapse two racks into us-east-1c, because that breaks rack failure independence and may violate RF assumptions. After replacement, run repair/rebuild as appropriate and verify with nodetool status plus consistency tests. nodetool assassinate is a last resort because it can leave data inconsistent.
3. How would you prevent recurrence? Name three concrete configuration or process changes.
Read the worked answer
Use a StorageClass with volumeBindingMode: WaitForFirstConsumer so the PV is provisioned only after the scheduler has placed the pod, matching the PV zone to the pod's rack. Keep one node pool per zone with a minimum size of at least one node, and make it an explicit rule that zonal pools are never scaled to zero while bound PVCs exist.
Derive the rack label and the zone label from the same source, and add a CI or admission check that a CassandraDatacenter rack's nodeAffinity matches the PV topology. Add a PodDisruptionBudget and cluster-autoscaler safeguards so a drain cannot remove the last node in a zone. Finally, take backups before infrastructure changes and run restore drills; a 24-hour-old backup is not a recovery plan for a stateful dataset.