Rolling Updates That Pass Readiness and Still Time Out
A web Deployment runs 24 replicas across three zones with rollingUpdate: maxSurge: 25%, maxUnavailable: 0, and an HPA with minReplicas: 24, maxReplicas: 40, target CPU 70%. The readiness probe is GET /healthz on port 8080, periodSeconds: 5, initialDelaySeconds: 0, default failureThreshold. The application returns 200 from /healthz as soon as the HTTP listener binds, roughly 3 seconds after start. The JIT and cache warm-up takes another 45 seconds, and during that window the pod serves requests at p99 of about 4 seconds. The ingress has a 3-second upstream timeout. Every deploy produces a 90-second band of 5xx errors and a burst of HPA scale-ups. Separately, the nightly kubectl drain of a single node reports cannot evict pod as it would violate the pod's disruption budget and never completes.
1. Diagnose the drain hang. What is the correct PDB value for this Deployment, and how do you derive it rather than guessing?
Read the worked answer
The draining node reports cannot evict pod as it would violate the pod's disruption budget because the Deployment has a PodDisruptionBudget with minAvailable: 24 against 24 replicas. Allowed disruptions = current ready replicas - minAvailable = 0. The eviction API refuses every request, so kubectl drain retries indefinitely and appears to hang. It is not a kubelet or CNI problem.
Derive the number from node topology, not from a round percentage. Ask: how many replicas of this Deployment live on one node at steady state? With 24 replicas over 9 nodes, that is 3 per node, so the budget must tolerate at least 3 disruptions simultaneously, plus one for safety during a rolling node upgrade: minAvailable: 20 (or maxUnavailable: 17%).
Check two failure modes of the fix. Too tight and drains hang again on a skewed node. Too loose and a single node loss drops capacity below what the HPA can absorb in its scale-up window -- so cross-check maxUnavailable against the HPA's measured response time, and remember the PDB only governs voluntary disruptions; it does nothing for hardware failure. Also confirm no other PDB in the same namespace targets the same pods, since budgets compose by intersection.
2. Redesign the probe configuration so that a rolling update does not send traffic to warming pods. State the numeric budgets and the risk of each choice.
Read the worked answer
The defect is that readiness reports a condition (listener bound) that is not the condition traffic requires (cache warm). Split the probes:
startupProbe:GET /healthz,periodSeconds: 5,failureThreshold: 18-- a 90-second budget, twice the observed 45-second warm-up. This prevents the liveness probe from killing a pod that is slow rather than broken, which is the usual reason teams raiseinitialDelaySecondsand then still fail on a slow node.readinessProbe: a new/readyzendpoint that returns 200 only after warm-up finishes,periodSeconds: 3,failureThreshold: 2. Endpoint removal and traffic starts only when the pod can actually serve.livenessProbe: keep it slow and forgiving (failureThreshold: 6,periodSeconds: 10) so it detects deadlocks, not slowness.
Risks: an overly generous startup budget delays detection of pods that never start (a broken image now looks healthy for 90 seconds). An overly tight readiness threshold flaps pods out of endpoints during GC pauses, which produces intermittent 503s that are hard to reproduce. Instrument the first-request latency of new pods and set the budgets from that distribution, then re-check the ingress timeout: if warm pods can still take 4 seconds, the 3-second ingress timeout remains a latent user-visible failure.
3. Explain the interaction between the HPA, unready pods, and `maxSurge: 25%`. Is there a capacity risk during deploys?
Read the worked answer
The HPA computes CPU utilisation over pods it considers ready; unready pods are excluded from the average (and newly created pods are ignored until they report metrics). During this rollout, every new pod is ready in 3 seconds but useless for 45, so it enters the denominator while contributing almost no CPU. The average collapses, the HPA scales up, more cold pods are created, the average stays low -- a self-reinforcing scale-up that ends only at maxReplicas: 40.
The capacity risk is arithmetic: 40 replicas at max, plus maxSurge: 25% of the Deployment's replicas during a rollout, can momentarily request around 46 pods. If a ResourceQuota or the node pool caps at 40, the surge pods go Pending and the rollout stalls at Waiting for rollout to finish while the old ReplicaSet is held open -- a state that can persist for the full progressDeadlineSeconds window. Confirm cluster headroom for maxReplicas + maxSurge before tuning either value.
Mitigations: fix readiness first (it removes the false signal), set HPA behavior.scaleUp.stabilizationWindowSeconds and a scaleDown policy, and consider scaling on a metric that reflects warm capacity such as requests per second per ready pod rather than raw CPU. Verify with kubectl get hpa -w during the next deploy rather than trusting the pre-change graph.