Containing a runaway ingest service without losing evidence
On a 48-core, 256 GB RHEL 9 host, ingest.service processes market data and spawns up to 64 worker processes. Twice this week the host became sluggish; dmesg showed the kernel OOM killer terminating sshd and chronyd instead of ingest, and journalctl -u ingest contains only the first 90 seconds of each incident. The unit currently has MemoryLimit=32G and TasksMax=infinity. The host uses cgroup v2 and systemd 252. The service must stay available during business hours, but one runaway worker must not take down the host. You cannot reboot, and audit records must remain on disk for 30 days. The root filesystem has 18 GB free; /var/log/journal is not persistent, and journald is using its default rate limit. The application owner wants evidence before approving cgroup changes, and the security team will not accept disabling the OOM killer.
1. The owner asks why MemoryLimit=32G did not contain the service. What would you inspect, and what cgroup settings would you apply to keep the host responsive while preserving the service?
Read the worked answer
Do not assume the configured limit is the effective limit. First run systemctl show ingest -p MemoryCurrent -p MemoryMax -p MemoryHigh -p MemoryLimit -p TasksCurrent -p TasksMax -p Delegate and inspect /sys/fs/cgroup/system.slice/ingest.service/memory.events, memory.current, and memory.stat. Use systemd-cgtop --order=memory and ps -eo pid,ppid,rss,comm --sort=-rss | head to identify the worker growth pattern. Check journalctl -k --grep='Out of memory|Killed process' to confirm which cgroup was charged for the OOM kills.
On cgroup v2, treat MemoryLimit= as suspect and set explicit controls: MemoryHigh=24G to force reclaim before hard failure, MemoryMax=30G as a hard cgroup cap, MemorySwapMax=0 to prevent swap thrash, and TasksMax=96 to stop fork storms. If the unit is restarted automatically, OOMPolicy=stop plus Restart=on-failure and a short RestartSec= turns a host-wide OOM into a contained unit restart. If the business cannot tolerate even that restart, OOMPolicy=continue is defensible only with application-level worker supervision and proof that leaked workers are reaped. Apply with systemctl daemon-reload and a controlled systemctl restart, not a reboot.
2. How would you make journald persistent and retain 30 days without filling the 18 GB root filesystem, and what evidence would you collect before the next incident?
Read the worked answer
Create /var/log/journal, run systemd-tmpfiles --create --prefix /var/log/journal, and set Storage=persistent in journald.conf. With only 18 GB free, do not set SystemMaxUse to the whole disk: reserve headroom for package and audit writes. A defensible starting point is SystemMaxUse=12G, SystemMaxFileSize=256M, MaxRetentionSec=30day, RateLimitIntervalSec=30s, and RateLimitBurst=10000; the burst is high enough to keep the incident burst but still bounded. Reload journald and verify with journalctl --disk-usage. If 12 GB cannot hold 30 days at the observed rate, the requirement conflicts with the disk budget and must be escalated rather than silently dropping records.
Before the next event, collect a repeatable bundle: systemctl show ingest -p MemoryCurrent -p MemoryMax -p MemoryHigh -p TasksCurrent, the service cgroup memory.events and memory.stat, systemd-cgtop --order=memory snapshots, ps sorted by RSS, dmesg -T, journalctl -k -b, and auditctl -l/audit logs if process execution auditing is enabled. Store the bundle outside the service cgroup and verify it survives a unit restart.
3. After applying limits, the service logs 'Cannot allocate memory' but the host stays responsive. Is that success? What would you check next?
Read the worked answer
It is partial success: the blast radius is now contained, but the service is not healthy. Check whether memory.events shows max or oom_kill increments and whether systemctl show -p Result reports oom-kill or a clean exit. Compare MemoryCurrent against MemoryHigh and MemoryMax; if the service is pinned at MemoryHigh and reclaiming constantly, it is being throttled rather than capped.
Next, separate an application leak from an undersized limit. Look at worker RSS over time, connection counts, queue depth, and whether TasksMax is being hit. If the app legitimately needs more memory, raise MemoryMax only with a measured headroom plan, not by removing the cap. If one worker leaks, fix or recycle that worker; do not lower the whole unit below its steady-state working set. Re-run the same evidence collection after the change so the next review compares like with like.