Intermittent Freezes on a 4 GB Web Server
Acme Retail runs its storefront on a cloud VM with 2 vCPUs, 4 GB RAM, and a 40 GB SSD. The stack is Nginx, PHP-FPM with 8 workers, and MySQL 8. During the daily peak from 10:00 to 11:00, the site freezes for 2–5 minutes. dmesg shows Out of memory: Killed process 1234 (mysqld). free -m during a peak reports total 3950 MB, used 3800 MB, free 50 MB, available 120 MB, and no swap is configured. vmstat shows zero swap-in/swap-out because no swap exists. The junior admin proposes creating an 8 GB swap file on the same SSD and setting vm.swappiness=60. The manager asks whether this will fix the problem or merely hide it, and what else should be done before the next peak.
1. Explain why the kernel's OOM killer selected mysqld and what the `available` value in `free -m` tells you that `free` does not.
Read the worked answer
The kernel overcommits virtual memory, so a process can request more memory than is physically available. When the system runs out of reclaimable pages and has no swap, the OOM killer selects a victim using oom_score, which rises with resident memory and can be adjusted via oom_score_adj. MySQL is likely the largest resident process, so it is killed to keep the kernel and other services alive.
free shows strictly unused RAM, while available estimates how much memory can be allocated to new processes without swapping, including reclaimable page cache. Here available is only 120 MB, so the system is in severe memory pressure. The OOM kill is a symptom, not the root cause; the root cause is that the workload exceeds 4 GB RAM at peak.
2. Evaluate the proposal to add 8 GB swap on the same SSD and set swappiness to 60. What would you recommend instead, and which commands would make the change persistent?
Read the worked answer
Swap on the same SSD will stop the immediate OOM kill, but it can turn a fast failure into slow thrashing: database pages get evicted to disk, and query latency spikes. vm.swappiness=60 is the default and encourages swapping, which is the opposite of what a database server usually wants. Recommend: (1) right-size the workload-reduce PHP-FPM workers from 8 to 4–6 and lower MySQL innodb_buffer_pool_size to fit; (2) add RAM if possible; (3) add a 2–4 GB swap file as a safety net, not as capacity, and set vm.swappiness=10; (4) monitor available and OOM events.
Commands:
fallocate -l 4G /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
echo '/swapfile none swap sw 0 0' >> /etc/fstab
echo 'vm.swappiness=10' > /etc/sysctl.d/99-swap.conf
sysctl --system
swapon --show
This makes the swap persistent and reduces the tendency to swap.
3. Which commands would you run to identify the top memory consumers and to confirm swap is active after configuration?
Read the worked answer
Use ps aux --sort=-%mem | head -n 15 to see top RSS consumers; top -o %MEM for live view; smem -r if installed for PSS. Check MySQL with mysqladmin status or SHOW ENGINE INNODB STATUS. After configuring swap, verify with free -h, swapon --show, and cat /proc/swaps. Also check dmesg | grep -i oom after the next peak. RSS is resident memory; VSZ is virtual size and can be misleading.