Pod Eviction Storm During Node Drain
A cluster has 30 nodes and 400 pods. A node pool upgrade requires draining 6 nodes one at a time. The team runs kubectl drain with default flags on the first node. Within minutes, 40 pods across the cluster go Pending, and several stateful workloads (a 3-replica database and a 2-replica message broker) lose quorum. Investigation shows: (a) the database pods have no PodDisruptionBudget; (b) the broker has a PDB with minAvailable: 2 but the drain ignored it; (c) many Pending pods are because the drained node's pods don't fit on remaining nodes (each pod requests 2 CPU, nodes have 8 CPU with 6 already used); (d) some pods were evicted but their PVCs are on a node-local volume.
1. Explain why the broker's PDB did not prevent the quorum loss, and why the database lost quorum.
Read the worked answer
Broker PDB: A PDB with minAvailable: 2 on a 2-replica broker means both pods must stay available - which is effectively maxUnavailable: 0. That should block eviction of either pod. But kubectl drain only respects PDBs for pods it evicts via the Eviction API; if the drain used --force or --disable-eviction, or if the pods were deleted directly, the PDB is bypassed. Also, if the broker pods are not matched by the PDB's selector (e.g., label mismatch), the PDB protects nothing. The likely cause: the PDB selector doesn't match the broker pods, or the drain bypassed eviction. Either way, a 2-replica broker with minAvailable: 2 is fragile - there's no headroom for any voluntary disruption, so the correct config is 3 replicas with minAvailable: 2.
Database: With no PDB, the drain evicted database pods freely. A 3-replica database losing 2 pods (or the wrong one) drops below quorum. The absence of a PDB is the direct cause; the fix is a PDB plus ensuring the drain respects it.
2. Diagnose the Pending pods and the PVC problem, and give a safe drain procedure.
Read the worked answer
Pending pods: Each drained pod requests 2 CPU. Remaining nodes have 8 CPU with 6 used → 2 CPU free each, so a 2-CPU pod just fits, but scheduling fragmentation and anti-affinity rules mean some don't. Draining 6 nodes at once would need 6×(pods per node) capacity; doing it one at a time still needs headroom for the drained node's pods. The cluster is over-committed relative to request size. Fixes: add a node (or use cluster-autoscaler) before draining, or reduce requests, or cordon-and-drain one node and wait for pods to reschedule before the next.
PVC problem: Node-local volumes (local PVs or hostPath) bind pods to a specific node. Draining that node makes the pod unschedulable elsewhere because the PVC is pinned. These workloads need either a node affinity that keeps them on a surviving node with the same local volume, or migration to network-attached storage. For stateful sets with local volumes, the drain must be done with the workload's own migration procedure, not kubectl drain.
Safe procedure: (1) Add/verify PDBs for all stateful workloads (broker: 3 replicas, minAvailable: 2; database: minAvailable: 2). (2) Verify PDB selectors match pod labels. (3) Ensure autoscaler headroom or add a node. (4) Drain one node with --ignore-daemonsets --delete-emptydir-data and without --force/--disable-eviction, then wait for all evicted pods to be Running and Ready before draining the next. (5) For local-volume pods, migrate via the app's own procedure. (6) Monitor kubectl get pdb throughout.