Stalled Rollout with a PDB That Is Not the Blocker

Arithmetic: maxSurge: 25% of 12 rounds up to 3, so at most 15 pods; maxUnavailable: 25% of 12 rounds down to 3, so at least 9 must stay Ready. The observed 15 pods / 9 Ready is exactly the configured envelope - the Deployment controller has done precisely what it was told and cannot proceed further. Nothing is "stuck" in the controller sense.

The PDB is not the cause. A PodDisruptionBudget gates the eviction API only - node drains, kubectl drain, cluster-autoscaler scale-down. The Deployment controller deletes ReplicaSet pods directly and never calls evictions. So minAvailable: 10 has zero effect on this rollout, and the fact that the rollout is running at 9 Ready (below 10) proves it.

That is the trap: the operator sees the PDB and assumes it is enforcing a floor. It is not, and relying on it for rollout safety is a real availability bug - here the PDB promises 10 but the rollout strategy quietly allows 9.

Real cause: the v2 pods never become Ready. The readiness budget is 20s + 3 x 10s = 50s before the kubelet marks the container NotReady, but the app needs ~75s. Every new pod fails readiness, so the controller never scales the old ReplicaSet down.

Immediate: kubectl rollout undo deploy/api (or --to-revision=N). This is capacity-safe to run now: the old ReplicaSet still exists with 9+ Ready pods, and the undo creates pods from the previous template, so the controller scales Ready capacity up before removing failing v2 pods. Watch kubectl rollout status deploy/api. Do not kubectl delete pod on the unready v2 pods - the controller recreates them from the same broken template and you churn the image pull for nothing, while briefly dropping below the envelope.

Durable fix - add a startup probe, which is exactly the primitive for "slow to become healthy once":

startupProbe:
  httpGet: {path: /healthz, port: 8080}
  periodSeconds: 10
  failureThreshold: 15      # 150s budget, ~2x the 75s worst case
readinessProbe:
  httpGet: {path: /healthz, port: 8080}
  periodSeconds: 5
  failureThreshold: 2
livenessProbe:
  httpGet: {path: /healthz, port: 8080}
  periodSeconds: 10
  failureThreshold: 3

While startupProbe has not succeeded, readiness and liveness are suppressed - so a slow warm-up no longer looks like a failure, and there is no need to inflate initialDelaySeconds on the other two (which would slow every subsequent restart).

Close the PDB/strategy gap: set maxUnavailable: 0 and keep maxSurge: 3. The rollout then never dips below 12 Ready, which is what the PDB was believed to guarantee. Cost: peak 15 pods worth of nodes and quota, plus the rollout cannot finish if the cluster lacks room for 3 extra pods. If capacity is tight, lower the PDB to minAvailable: 9 so the stated promise matches reality instead of being fiction.

In this order, because each one falsifies a different hypothesis:

# 1. Which revision is progressing, and how far
kubectl rollout status deploy/api --timeout=5s
#   'Waiting for deployment "api" rollout to finish: 3 out of 12 new replicas have been updated'

# 2. Per-ReplicaSet readiness - separates old (Ready) from new (not)
kubectl get rs -l app=api -o wide

# 3. The authoritative condition on one new pod
kubectl describe pod <v2-pod> | sed -n '/Conditions:/,/Events:/p'

Read these fields specifically:

• status.containerStatuses[].ready: false with state.running - the container is up and the probe is failing. state.waiting.reason: CrashLoopBackOff would mean it is not a probe issue at all.
• Event Readiness probe failed: HTTP probe failed with statuscode: 503 - the app is reachable and reporting not-ready, which is the warm-up. A connection refused would instead point at a wrong port in the probe or the app not listening yet.
• kubectl top pod <v2-pod>: 0m CPU (or heavily throttled under a low limits.cpu) with 503s rules out CPU starvation as the reason the cache never finishes.
• kubectl get pod <v2-pod> -o jsonpath='{.spec.containers[0].image}' plus the ReplicaSet hash confirms the image actually rolled out and you are not debugging a cached IfNotPresent tag.

Patching the Deployment's probes is a spec change, not an image change, so certification is untouched:

kubectl patch deploy/api --type=strategic -p '{"spec":{"template":{"spec":{"containers":[{"name":"api","startupProbe":{"httpGet":{"path":"/healthz","port":8080},"periodSeconds":10,"failureThreshold":15}}]}}}}}'
kubectl rollout status deploy/api

A strategic merge patch on containers merges by name, so startupProbe is added without dropping the existing readinessProbe, livenessProbe, or resources.

Two things to verify while it rolls:

1. Quota headroom - the new startupProbe means v2 pods stay up and warming for up to 150s instead of being churned, so you will sit at 15 pods longer. A ResourceQuota on pods or requests.cpu will reject the extra pods and the rollout hangs with ReplicaFailure events on the ReplicaSet; check kubectl get rs <new-rs> -o yaml for that before assuming the probe fix failed.
2. Capacity math with maxUnavailable: 0 - you need room for 3 extra pods for the full duration. If the cluster cannot schedule them, the rollout blocks with 0/12 updated and Pending pods, which is a scheduling failure and not a probe failure.

If scheduling headroom genuinely does not exist, temporarily set maxSurge: 1, maxUnavailable: 0 - slower, but it still never drops below 12 Ready.