CKA (CNCF): Rollout Stuck at 14 of 15 Pods, and the PDB Is Not the Reason

Capacity, not policy.

allocatable for app pods = 4 nodes x (4000m - 300m DaemonSet) = 14,800m
pods the rollout wants   = 12 old + ceil(12 x 25%) = 12 + 3 = 15
cpu those pods need      = 15 x 1000m = 15,000m
shortfall                = 200m -> exactly one pod cannot fit

Confirm it directly rather than trusting the story:

kubectl describe pod <pending-pod> | sed -n '/Events/,$p'
kubectl get nodes -o custom-columns=\
  NAME:.metadata.name,CPU:.status.allocatable.cpu
kubectl describe node worker-1 | sed -n '/Allocated resources/,/Events/p'

The scheduler is telling the truth: 14 pods fit, the 15th does not.

Why the PDB is innocent: a PodDisruptionBudget gates the Eviction API only - kubectl drain, kubectl drain-style node maintenance, and anything calling /eviction. A Deployment's rolling update deletes pods through the ReplicaSet, which is not an eviction, so minAvailable: 11 never enters the decision. The lead saw a stuck rollout and a PDB in the same namespace and drew the obvious wrong conclusion.

The topology spread constraint is also innocent here - 14 pods split 7/7 across two zones satisfies maxSkew: 1. It would bite later if you pinned a node.

Set maxSurge to an absolute 2, keep maxUnavailable: 0.

kubectl patch deployment api --type=merge -p \
  '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":2,"maxUnavailable":0}}}}'

Why 2 works: 12 + 2 = 14 pods x 1000m = 14,000m against 14,800m of room. Every time one of the new pods passes readiness, the ReplicaSet terminates one old pod, which frees 1000m for the next surge pod. The rollout walks through in pairs and never dips below 12 available.

Why not leave it at 25%: percentages round up, so 25% of 12 is 3, and 3 is precisely the number that does not fit. A surge of 2 with maxUnavailable: 0 is the correct answer here.

The alternative - maxSurge: 0, maxUnavailable: 1 - also completes the rollout, but it drops you to 11 available at the moment the old pod dies and the new one is not yet Ready. That is allowed by your PDB but violates the stated 12-available requirement, so it is the wrong answer for this question even though it is the right answer on a cluster with no headroom.

Note the Pending pod from the failed rollout will be cleaned up by the new ReplicaSet; no manual deletion is needed.

The cordon removed capacity, which made the PDB impossible to satisfy.

capacity with worker-2 cordoned = 3 nodes x (4000m - 300m) = 11,100m
api pods that fit              = floor(11,100m / 1000m) = 11
PDB minAvailable               = 11

Walk the drain:

1. worker-2 holds 3 api pods. The first eviction is allowed (disruptionsAllowed = 12 - 11 = 1).
2. The ReplicaSet tries to replace that pod - but no node has 1000m free, so the replacement sits Pending.
3. Available is now 11. disruptionsAllowed is 0, so the second eviction returns Cannot evict pod as it would violate the pod's disruption budget - forever.

The PDB is doing its job. The mistake was cordoning a node without checking that the remaining cluster could still host minAvailable.

Unblock, in order of preference:

• Add capacity first (a new node, or temporarily raise the node size). Then kubectl drain worker-2 --ignore-daemonsets --delete-emptydir-data finishes.
• Shrink the request for the duration (kubectl patch deployment api to 800m) so 11,100m holds 13 pods. Restore it after the drain.
• Deliberately lower the PDB with kubectl patch pdb <name> -p '{"spec":{"minAvailable":9}}', accept reduced resilience for the maintenance window, and put it back the moment the node is uncordoned. This is a conscious SLA decision, not a fix.

Inspect before acting - kubectl get pdb -n <ns> shows ALLOWED DISRUPTIONS, and kubectl get events will show the eviction failures. Also remember a pod in Terminating still holds its request until it is actually gone, so a drain can look stuck for a few seconds after the last eviction is granted.