OOMKilled at 980 MiB With a 1 GiB Limit
Contoso runs Kubernetes 1.28 on 60 nodes. The checkout-api is a Java 17 Spring Boot service with 24 replicas, a CPU request of 500m, a memory request of 512Mi, and a memory limit of 1Gi. Pods are OOMKilled 6 to 10 times per day, mostly around the 14:00 traffic peak. CPU stays below 40%, and the horizontal pod autoscaler targets 65% CPU, so it rarely scales. The JVM uses the default MaxRAMPercentage, heap dumps show about 250MiB live set, and container working set reaches 980MiB before termination. The service uses Netty direct buffers and creates many short-lived threads. Node allocatable memory is 58Gi, and the cluster has 14Gi free at peak. A pod disruption budget requires minAvailable 20. Rolling updates use maxUnavailable 1 and have caused brief 5xx spikes.
1. Diagnose why the pods are OOMKilled even though the Java heap live set is only about 250MiB.
Read the worked answer
The OOMKill is a cgroup-level event, not a Java heap OutOfMemoryError. With a 1Gi limit and the default MaxRAMPercentage of 25%, the heap tops out near 256MiB, which matches the 250MiB live set. The remaining roughly 700MiB is off-heap: Netty direct buffers, Metaspace, code cache, thread stacks, GC structures, and memory-mapped files. The working set of 980MiB just before termination supports that. The HPA misses it because it watches CPU, and this is a memory-driven failure. Diagnose with container_memory_working_set_bytes, kube_pod_container_status_terminated_reason, JVM Native Memory Tracking, and a thread count. Set explicit -XX:MaxRAMPercentage=70, -XX:MaxMetaspaceSize, and -XX:MaxDirectMemorySize so the JVM budget is visible, then raise the limit to 1.5Gi with a request of 1Gi. Do not set the request equal to the old limit without measuring; that would reduce bin-packing efficiency.
2. Recommend requests, limits, and autoscaling changes. What tradeoffs are you accepting?
Read the worked answer
Requests should reflect the scheduler guarantee, not the peak. Use a 1Gi request and a 1.5Gi limit for checkout-api; with 24 replicas that is 36Gi reserved, which fits the cluster's allocatable memory but should be checked for node fragmentation. Keep the CPU request at 500m and add a CPU limit only if you have measured throttling; otherwise rely on requests. Change the HPA to scale on memory working set and a custom requests-per-second metric, with a three-minute stabilization window to avoid flapping. Set behavior to scale up quickly and scale down slowly. With minAvailable 20 and 24 replicas, only four pods can be disrupted, so a rolling update with maxUnavailable 1 is safe, but add a readiness gate that waits for JVM warmup. The tradeoff is higher reserved memory and lower pod density in exchange for fewer OOMKills. Test the new limits in a canary namespace under a synthetic 14:00 peak before production.
3. How would you roll out the change safely given the existing 5xx spikes during updates?
Read the worked answer
Use a canary deployment with 10% of traffic, not a full rolling update. Monitor OOMKilled count, working set, p99 latency, and 5xx rate for at least 30 minutes at peak. Automate rollback if OOMKills exceed zero or p99 rises more than 20% above baseline. Set terminationGracePeriodSeconds to 30 and a preStop sleep so in-flight requests drain; otherwise the 5xx spikes will continue. Because the PDB allows only four disruptions, scale up temporarily before the rollout to keep capacity. Run a load test against the canary that reproduces the 14:00 pattern, including Netty buffer usage. If the canary passes, promote in 25% steps. Document the JVM flags and limits in the deployment manifest so future changes do not regress. Do not rely on kubectl rollout undo alone; it does not fix a bad memory limit if the old ReplicaSet had the same limit.