Investigating Intermittent 504s After a Dependency Upgrade
After upgrading an HTTP client library, a checkout service reports intermittent 504 responses from its gateway. The errors occur on about 2% of requests, clustered in bursts of 30-90 seconds, roughly every 20 minutes. Server-side logs show the checkout service completing the request in 150-400 ms, well under the gateway's 5 s timeout. The client library's changelog mentions a new default connection pool size of 10 and a new default idle timeout of 15 seconds. The service makes about 120 downstream calls per second at peak, each taking 80 ms at p50. Nothing else changed. The team wants to roll back the library.
1. Assess whether the gateway timeout explains the 504s, and form a hypothesis that fits the evidence.
Read the worked answer
The gateway timeout does not explain them. The service reports completing in 150-400 ms against a 5 s limit, so the request is not slow at the application layer. A 504 generated by the gateway means the gateway did not receive a complete response in time, which can happen even when the application thinks it finished, if the response is stuck in a connection or the request never reached the app.
The evidence points to connection pool exhaustion. At 120 calls/second with 80 ms p50 latency, required concurrency is 120 x 0.08 = 9.6, so a pool of 10 is right at the edge at median. Any latency spike above p50 immediately exceeds 10 concurrent calls, and requests queue for a connection. The 15-second idle timeout adds churn: connections idle out and must be re-established, and during reconnection bursts the pool is briefly smaller. That fits the periodic 20-minute bursts and the 30-90 second duration better than a timeout.
So the hypothesis is: the new defaults (pool of 10, 15 s idle) are undersized for 120 calls/second, causing queueing and occasional gateway-visible stalls. This is testable: raise the pool size and idle timeout in a staging environment and see if the bursts disappear.
2. The team wants to roll back immediately. Argue for or against, and describe the verification you would run before closing the incident.
Read the worked answer
Rollback is defensible as a stabilization step because it is fast and the change is isolated, but it is not a diagnosis. If the root cause is capacity, rollback restores the old defaults and the problem disappears without anyone learning the safe operating envelope, so the next upgrade repeats it.
A better sequence: roll back to restore service, then reproduce in staging with the new library and the same traffic profile. Verify by:
- Measuring pool wait time and connection acquisition failures as first-class metrics. If wait time rises before each burst, the hypothesis is confirmed.
- Running a load test at 120 calls/second with pool sizes of 10, 20, and 40, and idle timeouts of 15 s and 60 s, recording p99 latency and 504 rate for each.
- Checking whether bursts align with connection re-establishment, e.g. a spike in new TCP connections every 20 minutes.
- Confirming the gateway's own logs show the 504 originating from an incomplete upstream response rather than a slow application response.
Close the incident only when the chosen settings have headroom at 2x peak, the settings are pinned explicitly rather than inherited from library defaults, and a regression test fails if pool size drops below the computed concurrency requirement.