Drain Blocked by a PodDisruptionBudget and a NotReady Replica

The PDB does not count desired replicas; it counts healthy pods. api has 4 pods, but one on worker-1 is Running with 0/1 Ready, so currentHealthy is 3. desiredHealthy is minAvailable: 3. The controller computes disruptionsAllowed = currentHealthy - desiredHealthy = 3 - 3 = 0. kubectl get pdb api-pdb therefore shows ALLOWED DISRUPTIONS 0, and kubectl describe pdb api-pdb records eviction-denied events. Drain cordons the node and then uses the Eviction API for each pod; every eviction of an api pod is rejected because it would leave only 2 healthy replicas. Drain retries and appears to hang.

The NotReady pod is the trigger, not the PDB value. Had all 4 pods been Ready, allowed disruptions would be 1, and drain could evict one api pod at a time, waiting for its replacement to become Ready before the next. The second blocker is indexer-0: it is a bare pod with a local-path PVC, so it has no controller to recreate it and drain will refuse to evict it without --force. --ignore-daemonsets does not address either issue.

1. Abort the hung drain with Ctrl-C. The node remains cordoned. If capacity is tight, temporarily uncordon it until readiness is fixed, then cordon again; otherwise evicted pods may have nowhere to schedule.
2. Diagnose the NotReady pod: kubectl describe pod <pod>, kubectl logs <pod> --previous, and kubectl get cm api-config -o yaml. If a bad ConfigMap update broke the probe, restore the ConfigMap and kubectl rollout restart deploy/api, or kubectl rollout undo deploy/api. Wait for kubectl rollout status deploy/api and confirm 4/4 Ready.
3. Confirm kubectl get pdb api-pdb now shows 1 allowed disruption and kubectl get pods -o wide shows all api pods Ready.
4. Check capacity on the remaining nodes with kubectl describe nodes and verify that worker-2's non-DaemonSet pods fit elsewhere. With no autoscaler, scale down non-critical workloads or add a node if they do not.
5. Handle indexer-0: inspect kubectl get pod indexer-0 -o yaml. If it has no ownerReference and its PVC is node-local, decide explicitly whether to back up and recreate it on another node or accept downtime. Then kubectl drain worker-2 --ignore-daemonsets --delete-emptydir-data --timeout=20m.
6. If eviction is still blocked, scale api to 5 replicas so allowed disruptions becomes 2, or temporarily lower the PDB to minAvailable: 2 with change approval. --disable-eviction makes drain delete pods directly, bypassing the PDB entirely. That can remove the last healthy replica and turn a maintenance task into an outage; it is a last resort with explicit risk acceptance, not a first move.

Fix the readiness design first. A readiness probe that depends on a mutable ConfigMap turns a configuration change into a cluster-wide availability event. Use immutable ConfigMaps with a content hash in the name, so a bad update produces a new ReplicaSet and a controlled rollout instead of breaking running pods. Keep the PDB but make it match the replica count: with 4 replicas, maxUnavailable: 1 is equivalent to minAvailable: 3 and is less brittle if the replica count changes.

Give indexer-0 a controller and portable storage. A bare pod with a local-path PVC is inherently undrainable without force. Convert it to a StatefulSet backed by replicated storage, or document a tested migration procedure with backups. Add topology spread constraints or anti-affinity so all api replicas cannot concentrate on one node.

Add operational safeguards: alert when kube_poddisruptionbudget_status_disruptions_allowed is 0 and when any critical pod is NotReady; keep at least one node of spare capacity or install a cluster autoscaler; run kubectl drain --dry-run=server as a pre-check in the runbook; and rehearse the drain in staging. Finally, separate DaemonSet and local-storage pods from the eviction path explicitly in the runbook so the on-call engineer does not discover them at 02:00.