I85PREP
⌘ K
LCA ICT Specialism study materials
LCA ICT Specialism study resources

The 02:10 OOM Restart Loop on app-prod-03

LCA ICT Specialism · Put your knowledge to work in context.

All case studies

The 02:10 OOM Restart Loop on app-prod-03

A monitoring alert fires at 02:10 for app-prod-03, a 4 vCPU / 8 GB Ubuntu 22.04 host running a Java API behind nginx. Load average is 38, but CPU idle is 62%. free -m shows 7.6 GB used and 40 MB available, and swap is fully consumed. systemctl status api shows 14 restarts in the last hour with status 137, and dmesg contains Out of memory: Killed process 28411 (java). Grafana shows memory climbing steadily since the 18:00 deploy while request rate is flat. The on-call engineer proposes adding Restart=always, creating 4 GB of swap, and closing the incident. The unit file has no MemoryMax setting and the JVM starts with -Xmx6g. The same host runs a log shipper and node_exporter. Business hours begin in six hours and the API is customer-facing.

1. Interpret the evidence: what failure mode do the load average, idle CPU, exhausted swap and exit status 137 point to, and what would you check next to confirm it?

Read the worked answer

The combination points to memory exhaustion and reclaim stalls, not a CPU problem. Linux load average counts runnable tasks plus tasks in uninterruptible sleep (D state). With 62% idle, the CPUs are waiting on something - almost certainly page reclaim and swap I/O. Status 137 is 128 + 9, meaning SIGKILL; the kernel OOM killer sends SIGKILL, and dmesg confirms the java process was the victim. free -m showing 40 MB available with swap fully consumed means the working set no longer fits in RAM.

Confirm with:

  • vmstat 1 10: look for high wa and sustained si/so. If swap in/out is near zero but available memory is low, the pressure may be a spike rather than ongoing thrash.
  • ps -eo pid,stat,rss,comm --sort=-rss | head: D-state processes and the real RSS of java, the log shipper and node_exporter.
  • journalctl -k --since -2h | grep -i -E 'oom|killed process': the full OOM report shows the cgroup and memory statistics.
  • systemctl show api -p NRestarts -p ExecMainStatus and systemd-cgtop: confirm the restart loop and which cgroup holds memory.
  • jcmd <pid> VM.native_memory summary if enabled, or pmap -x: the heap is capped at 6 GB, but metaspace, thread stacks, code cache, direct buffers and GC structures add 1-2 GB.

Do not stop at the reflex that high load means CPU saturation; the evidence says otherwise.

2. Critique the proposed fix of adding `Restart=always`, creating 4 GB of swap, and closing the incident. What would you do in the six hours before business hours, given the API is customer-facing?

Read the worked answer

Restart=always changes nothing about the cause: systemd is already restarting the unit, and each restart loses in-flight requests and may hide a failing service from alerting. Adding 4 GB of swap to an 8 GB host with a 6 GB heap can make things worse: the JVM will thrash and GC pauses will grow, turning an OOM kill into a long latency incident. Closing the incident with an unexplained memory climb since the 18:00 deploy is not defensible.

In the next six hours, protect the host and restore headroom:

  1. Cap the service so it cannot take down the host: set MemoryHigh=5G, MemoryMax=6G and MemorySwapMax=0 on the api unit, then systemctl daemon-reload and restart in a controlled step. The kernel will throttle, then kill only the cgroup rather than the whole host.
  2. Reduce -Xmx6g to about -Xmx4g and set -XX:MaxMetaspaceSize=512m. A 6 GB heap on an 8 GB host leaves too little for the OS, page cache, the log shipper and JVM overhead.
  3. Check whether the 18:00 deploy changed heap behaviour. If a leak is suspected, take a heap dump before restarting and compare it with a baseline.
  4. Add a temporary alert on MemAvailable < 800 MB and on restart count, and tell the on-call team the service may still restart.

After the immediate risk is contained, do the root-cause work: heap analysis, a load test against the new limits, and a post-incident review. Do not treat swap as the primary fix on a JVM host.

3. What permanent configuration and monitoring changes would you make, and how would you verify they work?

Read the worked answer

Permanent changes should make the host survive a bad deploy and make the failure visible early.

Configuration:

  • Keep cgroup limits on the api unit (MemoryHigh, MemoryMax, MemorySwapMax=0) and add limits to the log shipper so a log burst cannot starve the API.
  • Set Restart=on-failure with RestartSec=5 and StartLimitIntervalSec/StartLimitBurst so a crash loop is rate-limited and alerts rather than spinning forever.
  • Use OOMScoreAdjust to protect sshd and node_exporter.
  • Size the heap from measured live-set data, not from total RAM, and document the arithmetic in the runbook.

Monitoring:

  • Alert on node_memory_MemAvailable_bytes below a threshold with a short for: clause, and on swap-in/out rate.
  • Alert on systemd_unit_state and NRestarts for api, and on cgroup memory approaching MemoryMax.
  • Add a dashboard panel for JVM heap versus RSS so off-heap growth is visible.

Verification: in staging, run a memory stress test such as stress-ng --vm alongside the API and confirm the host stays responsive, the cgroup kills only the API, and both alerts fire. Check systemctl show api -p MemoryMax and systemd-cgtop during the test. Finally, confirm that Restart=on-failure still recovers from a normal crash but stops after the burst limit, and record the observed thresholds in the post-incident review.