Intermittent 502s Under Load Traced to PHP Workers and Autoscaling
A Drupal site on Acquia Cloud has three web nodes in Prod. During a marketing push, traffic triples for about 40 minutes. Monitoring shows intermittent 502 Bad Gateway responses on roughly 3% of requests, spiking exactly during the push. Web node CPU is moderate (50–60%), but PHP-FPM worker saturation is high and request queueing is visible. The site has a known slow uncached endpoint (a JSON feed) that is hit heavily during the push and takes ~2 seconds per request. Varnish is in front, but the feed is uncached because it varies by a query parameter. Database latency is normal. The team's first instinct is to add more web nodes.
1. Diagnose the 502s and evaluate whether adding web nodes is the right first response.
Read the worked answer
502s with moderate CPU but saturated PHP-FPM workers and request queueing indicate the bottleneck is PHP worker capacity and request duration, not raw CPU or database. When all workers are busy on slow requests, the upstream (web server) can't get a worker in time and returns 502 to the client. The slow uncached feed at ~2s per request, multiplied by heavy traffic, is the amplifier: it ties up workers for long periods.
Adding web nodes is a plausible but incomplete response. It adds worker capacity, which can relieve saturation, but it does not address the root cause - a slow, uncacheable endpoint being hammered. It also scales cost linearly and may not help if the feed's cost is intrinsic (e.g., expensive query or serialization). The better first move is to reduce worker occupancy: make the feed cacheable or cheaper, and protect the origin from the traffic pattern.
2. Propose a mitigation plan that addresses both the immediate incident and the root cause.
Read the worked answer
Immediate:
- Make the feed cacheable: normalize or strip the varying query parameter where possible, or add a cache-varying strategy so Varnish can serve it. If the parameter truly changes the response, cache per-value with a short TTL and a bounded key space.
- If it can't be cached quickly, rate-limit or queue the feed at the edge, or temporarily serve a degraded/static version during the push.
- As a stopgap, add web nodes to increase worker capacity - but only alongside the above, and monitor worker saturation and 502 rate to confirm effect.
- Watch for a thundering herd when caches refill; stagger or warm caches.
Root cause:
- Profile the feed endpoint; reduce its ~2s cost (query optimization, precomputation, pagination limits).
- Set appropriate PHP-FPM worker limits and timeouts so slow requests fail fast rather than occupying workers.
- Add alerting on worker saturation and 502 rate, and load-test the feed before future pushes.
- Consider a separate pool or path for expensive endpoints so they can't starve the main site.
3. What monitoring signals would you use to confirm the fix and catch recurrence early?
Read the worked answer
Track PHP-FPM worker utilization and queue depth (the leading indicator), 502 rate (the symptom), and request latency percentiles for the feed specifically. After the fix, confirm worker saturation stays below a threshold and 502s return to zero during a controlled load test. Add alerts on worker saturation crossing a warning threshold and on 502 rate exceeding a small baseline, so the team is paged before customers see errors. Include the feed's cache hit ratio as a signal - a drop there predicts worker pressure.