Throughput Collapse After Raising Worker Concurrency
A monitoring service on a 16 GB Linux host runs 40 worker processes, each with a measured resident working set of 350 MB in its hot loop; the kernel and page cache together need about 3 GB to keep the log shipper healthy. Pages are 4 KB. At 20 workers the host sustained 85% CPU utilisation and 1,900 requests/s. After the concurrency limit was raised to 40, throughput fell to 700 requests/s, CPU utilisation dropped to 35%, the disk queue length sat at 30, and instrumentation reported 1,400 major faults per second with a mean fault service time of 8 ms. The team proposes three fixes: (i) switch the page-replacement policy from LRU to FIFO, (ii) lower the concurrency limit back to 20, (iii) enable 2 MB transparent huge pages so fewer TLB entries and page-table walks are needed. The manager wants to know which of these is actually a fix and which only hides the problem.
1. Show with arithmetic whether this is thrashing or a replacement-policy defect, and name the single measurement that settles it.
Read the worked answer
Frames: 16 GB / 4 KB = 4,194,304. Demand: 40 × 350 MB = 14,000 MB working sets + 3,000 MB for kernel and page cache = 17,000 MB = 17 GB against 16 GB - a deficit of ~1 GB, roughly 262,000 frames, about 6% short. Fault service demand: 1,400 faults/s × 8 ms = 11.2 seconds of disk service required per second of wall clock, i.e. the device is asked for 1,120% of its capacity. That is the decisive number: any system with a fault-service demand above 1.0 spends all its time in the fault path, so runnable processes are always blocked, CPU idles at 35%, and the queue grows to 30.
The pattern - utilisation falls while concurrency rises, throughput falls, disk queue rises - is the textbook signature of thrashing, not of a policy defect. Also note the throughput at 20 workers (1,900/s) versus 40 (700/s) is sub-linear in the wrong direction: adding concurrency removed useful work.
2. Assess proposal (i), the switch to FIFO, and proposal (ii), restoring concurrency 20. Support the second with numbers.
Read the worked answer
(i) FIFO: reject. FIFO lacks the stack property, so it is subject to Belady's anomaly - adding frames can increase faults. It also evicts by age rather than recency, so it will discard a worker's hot inner-loop page simply because it was loaded early, pushing the fault rate up. LRU and CLOCK are stack algorithms and monotone in the number of frames; replacing LRU with FIFO cannot create capacity and can only make the deficit more expensive.
(ii) Concurrency 20: accept as the immediate fix. Working sets become 20 × 350 MB = 7,000 MB, plus 3,000 MB = 10 GB of 16 GB, leaving 6 GB of headroom for bursty processes and page cache. Faults should fall to the compulsory/capacity floor (tens to low hundreds per second), fault-service demand drops well below 0.2, and CPU should recover to the 85% previously measured.
The durable fix is memory: 40 × 350 MB + 3 GB = 17 GB of steady-state demand; sizing to 22–24 GB gives the 25–30% headroom needed so that a burst does not re-enter the cliff.
3. Assess proposal (iii), 2 MB transparent huge pages.
Read the worked answer
The benefit is real but narrow: a 350 MB working set is 89,600 base pages versus 175 huge pages, so TLB reach and page-table-walk cost improve sharply - this is a latency fix for TLB misses, not a throughput fix for a memory deficit.
It can actively hurt here. Each huge-page fault brings in 2 MB; if the process consumes only 4 KB of it before eviction, the fault-service demand at 1,400 faults/s becomes ~2.8 GB/s of I/O instead of 11 MB/s, and the thrashing gets worse. Under sustained memory pressure, khugepaged frequently cannot obtain a contiguous 2 MB block at all, so the feature silently no-ops.
Verdict: treat it as a follow-up optimisation to be measured after the capacity problem is solved - reduce concurrency and/or add RAM first, then enable THP and re-measure faults/s and CPU. The manager's framing is right: (ii) fixes, (i) worsens, (iii) hides a symptom whose cause is elsewhere.