Deployment Rollout Stalled by Readiness Probe
A team runs a Deployment named checkout with 6 replicas, maxSurge: 1, maxUnavailable: 0, and minReadySeconds: 10. The readiness probe hits /healthz on port 8080 every 5s with a 2s timeout and failureThreshold: 3. After a new image rollout, kubectl rollout status hangs at '2 out of 6 updated'. The two new pods are Running but never Ready. Their logs show the app started and is serving on port 8080. kubectl describe pod shows readiness probe failures with 'connection refused'. A previous image worked. The new image's entrypoint now takes ~40s to bind the port because it runs a startup migration, but the liveness probe (same path, initialDelaySeconds: 5, failureThreshold: 3) is also configured.
1. Explain why the rollout is stuck and why the pod is not being killed despite liveness failures.
Read the worked answer
The new pods fail the readiness probe because the app doesn't bind port 8080 for ~40s, while the probe begins immediately and refuses connections. With maxUnavailable: 0 and maxSurge: 1, Kubernetes will not scale down old pods until new ones become Ready, so the rollout stalls at 2 updated.
The liveness probe should be killing the pod, but the key detail is that liveness failures only trigger a restart after the probe has been failing for failureThreshold consecutive checks. With initialDelaySeconds: 5 and a 2s timeout every 5s, liveness should fail around ~5-20s and restart the container - which would reset the 40s timer, causing a CrashLoopBackOff-like restart cycle rather than a clean stall. If the pod is not restarting, the most likely explanation is that the liveness probe is not actually configured against the same endpoint or is being masked. Either way, the diagnosis is a startup timing mismatch: probes fire before the app is listening. Verify with kubectl get pod -o yaml that both probes are present and match the actual path/port.
2. Propose a configuration change that fixes the rollout without weakening failure detection for genuinely broken pods.
Read the worked answer
Add a startupProbe on /healthz:8080 with failureThreshold: 12 and periodSeconds: 5 (allowing up to 60s). While the startup probe is running, liveness and readiness probes are disabled, so the slow migration won't trigger restarts or false readiness failures. Once startup succeeds, the existing liveness/readiness probes take over with their normal thresholds.
Alternatively, raise initialDelaySeconds on liveness/readiness to ~45s, but that permanently delays detection of real failures on every restart. The startupProbe is better because it only affects the initial boot window.
Also consider: keep maxUnavailable: 0 (safe) and confirm the migration is idempotent so a restart mid-migration doesn't corrupt state. After the fix, kubectl rollout status should progress 1 pod at a time, each waiting minReadySeconds: 10 before the next.