The Kubernetes Rollout That Sent Traffic to Cold Pods
A Kubernetes cluster runs a customer-facing API with 30 replicas behind a Service. HPA targets 60% CPU with min 10 and max 40. A PodDisruptionBudget sets minAvailable: 2. During a node upgrade, 4 of 12 nodes are drained sequentially. The deployment uses a rolling update with maxUnavailable: 25% and readiness probes hitting /health, which returns 200 as soon as the process starts but before the 90-second cache warm completes. The liveness probe also hits /health with initialDelaySeconds: 10. During a recent upgrade, users saw 5xx errors for 6 minutes, and some pods were OOMKilled after receiving traffic. The team wants to add a startup probe and change the PDB to maxUnavailable: 1. The API has a 200 ms p95 latency target and each pod warms a 2 GB in-memory cache.
1. Explain the failure chain that produced the 5xx errors and OOMKills. Which Kubernetes settings contributed, and which are red herrings?
Read the worked answer
The primary failure is that readiness did not represent serving capacity. /health returned 200 before the 90-second cache warm, so the Service sent traffic to pods that were alive but not ready in the functional sense. Those pods consumed CPU and memory warming a 2 GB cache while also serving requests, which explains the OOMKills and latency spikes. As CPU rose, the HPA scaled up and created more cold pods, amplifying the problem.
The rolling update with maxUnavailable: 25% allowed up to 7 of 30 pods to be unavailable at once, but that is a deployment setting and is not governed by the PDB. The PDB's minAvailable: 2 is far too weak for a 30-replica service, but it applies to voluntary evictions such as node drains, not to rolling updates. That makes it a contributing risk during the node upgrade, not the direct cause of the 5xx burst. The liveness probe's initialDelaySeconds: 10 is also a risk: it can restart a warming pod if /health becomes slow, but the more direct defect is that the readiness endpoint was not a true readiness check. The HPA target itself is not inherently wrong, but scaling on warm-up CPU can add cold replicas.
In short: premature readiness plus heavy warm-up plus traffic led to OOMKills; the PDB and liveness settings made the system less resilient but are not the primary explanation.
2. Evaluate the proposed startup probe and PDB change. What else is required, and how would you validate the fix before the next node upgrade?
Read the worked answer
A startup probe is the right direction, but only if it checks a startup-specific condition and allows enough time: failureThreshold multiplied by periodSeconds must exceed the 90-second warm-up, with margin. The readiness probe must move to a /ready endpoint that verifies cache warm and dependency health; otherwise the startup probe only delays liveness checks while readiness still admits traffic too early. Add a preStop hook and graceful shutdown so in-flight requests drain before a pod disappears, and set realistic CPU and memory requests and limits so the cache plus request load fits within the pod.
Changing the PDB to maxUnavailable: 1 is safer for node drains, but it is not a rollout control and can block maintenance if too strict. Prefer a PDB expressed as a high percentage, such as minAvailable: 90%, and consider unhealthyPodEvictionPolicy so unhealthy pods can be removed. For the deployment, use maxUnavailable: 1 or a small surge with a readiness gate, and add topology spread or anti-affinity so a node drain does not remove a large fraction of serving capacity.
Validate in staging with the same node count and cache size. Drain four nodes sequentially under synthetic load, observe 5xx rate, readiness failures, OOMKills, p95 latency, and pod restart counts, then run a canary in production with automated rollback. Do not rely on the startup probe alone; prove that no pod receives traffic before it can serve.