OOMKilled Pod With a Healthy Heap
A Java ingest service runs in a Kubernetes pod with a 8 GiB memory limit on a node with 128 GiB free and no node-level pressure. JVM flags are -Xmx6g -XX:MaxMetaspaceSize=512m; no MaxDirectMemorySize is set. The service uses Netty with pooled direct buffers (16 MB chunks, 16 arenas) to stream roughly 40 GB of records per nightly run. Around 20 minutes into each run the pod is killed with exit code 137. container_memory_working_set_bytes climbs steadily to 8 GiB; the heap, however, sits flat at about 4.2 GiB with 3% GC time and no OutOfMemoryError in the logs. A thread dump shows 200 live threads. The team's working theory is a heap leak and the proposal is to raise -Xmx to 8g.
1. Explain why the pod dies with exit code 137 instead of throwing a Java heap OutOfMemoryError, and why raising -Xmx would make things worse.
Read the worked answer
Exit code 137 means SIGKILL from the cgroup OOM killer, not a JVM allocation failure. The JVM throws OutOfMemoryError only when the heap cannot satisfy an allocation; here the heap is at 4.2 GiB against a 6 GiB ceiling, so the heap allocator never complains. The kernel, however, accounts total cgroup usage - anonymous memory from the heap, plus direct buffers, metaspace, code cache, thread stacks and any page cache - against the 8 GiB limit.
The arithmetic is consistent with the kill: heap 4.2 GiB + direct buffers ~3.6 GiB + metaspace 0.3 GiB + 200 thread stacks at 1 MB (0.2 GiB) + code cache 0.1 GiB ≈ 8.4 GiB, just over the limit. The direct-buffer growth is the missing term: with no MaxDirectMemorySize, the limit defaults to -Xmx, so the pool may legally grow to 6 GiB, and pooled Netty arenas rarely return chunks promptly.
Raising -Xmx to 8g would raise the cgroup ceiling on direct buffers to 8 GiB while leaving the pod limit at 8 GiB, so the kill would come sooner and the diagnosis would be further obscured. The heap is innocent and should not be the first thing changed.
2. Design the measurement that attributes memory to the right subsystem before any flag is changed.
Read the worked answer
The goal is to split cgroup usage into reclaimable and unreclaimable components and then attribute the anonymous portion.
From outside the container, read /sys/fs/cgroup/memory.stat (or the container's cgroup v2 equivalent) and compare anon, file, kernel_stack, sock and slab. If anon dominates while file stays small, this is not page cache and not reclaim pressure - it is committed memory.
From inside, enable Native Memory Tracking at startup (-XX:NativeMemoryTracking=summary) and run jcmd <pid> VM.native_memory summary at 5, 10 and 15 minutes. That separates Java heap, class, thread, code and - crucially - the Internal/direct-buffer category, giving a growth rate per subsystem.
Correlate with a heap histogram (jcmd GC.class_histogram) to confirm heap stability, and with a buffer-pool metric (Netty PooledByteBufAllocator metrics, or BufferPoolMXBean for direct buffers) to see whether the pool's chunk count rises monotonically. Sampling at intervals matters: a single snapshot cannot distinguish a steady state near the limit from a leak, and the kill is time-dependent.
Only after the growth is attributed should the limit be tuned; the measurement also tells you whether the nightly job's peak is intrinsic, which determines whether tuning or restructuring is the right answer.
3. Give a remediation plan with ranked options and the trade-off of each.
Read the worked answer
Option 1, bound the native memory: set -XX:MaxDirectMemorySize to a value derived from measured peak (e.g. 2g), keep -Xmx at 6g, and ensure total stays below the cgroup limit with headroom for stacks and metaspace - roughly 6 + 2 + 0.5 + 0.3 = 8.8 GiB, so the cgroup limit must also be raised, or the heap reduced to 4g. Trade-off: a too-tight direct limit converts an OOMKill into an OutOfMemoryError: Direct buffer memory under burst load, which is at least a clear failure.
Option 2, reduce the demand: cap Netty arenas and chunk size, and make buffer release explicit on the streaming path so the pool does not grow to steady-state worst case. Trade-off: fewer arenas can raise contention on allocation.
Option 3, sizing discipline: adopt -XX:MaxRAMPercentage with an explicit MaxDirectMemorySize fraction so pod limit changes propagate correctly, rather than hand-tuned absolute values.
Rank Option 1 and 2 together first - they address the cause. Raising the pod limit alone is the weakest option: it buys time proportional to the leak rate and will fail again on a larger nightly run. Whatever is chosen, add an alert on container_memory_working_set_bytes at 85% so the next regression is caught before the kill.