The Intermittent Checkout Outage
An online store's checkout service calls a payment provider over HTTPS. Roughly 0.4% of checkouts fail with a 30-second timeout, clustered in bursts of a few minutes, mostly at peak traffic. Retries are enabled: the client retries 3 times with no delay. Logs show the payment provider sometimes responds slowly (up to 8 seconds) but rarely errors. During bursts, the checkout service's own thread pool saturates and unrelated endpoints (product pages) also slow down. A developer proposes raising the retry count to 5 and increasing the timeout to 60 seconds to 'give the provider more room'. The team also has no circuit breaker and no per-request deadline propagated to the provider.
1. Diagnose the failure mode and evaluate the developer's proposal.
Read the worked answer
This is a retry storm amplifying a slow dependency. The provider occasionally takes up to 8 seconds. With a 30-second timeout and 3 immediate retries, each slow call occupies a worker thread for up to 30 seconds and may be multiplied by up to 4 attempts. Immediate retries add load exactly when the provider is already slow, so a brief slowdown becomes a burst of queued requests. The shared thread pool then saturates, which is why product pages -- an unrelated endpoint -- also degrade. That cross-endpoint slowdown is the tell that the problem is resource exhaustion in the caller, not the provider alone.
The proposal makes it worse. Raising retries to 5 increases amplification by roughly 67% per failing call. Raising the timeout to 60 seconds means slow calls hold threads twice as long, so saturation arrives sooner. 'Giving the provider more room' is backwards: the caller is the bottleneck, and the provider is being pushed harder.
Root causes to address:
- No retry budget or backoff -- retries are unbounded in aggregate and synchronized.
- No circuit breaker -- the caller keeps hammering a degraded dependency.
- No deadline propagation -- the provider does not know the caller has given up, so it keeps working on abandoned requests.
- Shared thread pool couples checkout to unrelated endpoints.
Fixes: exponential backoff with jitter, a retry budget (e.g. retries capped as a fraction of total requests), a circuit breaker that fails fast after an error/latency threshold, a per-request deadline passed to the provider, and bulkheads or a separate pool so checkout cannot starve product pages. Timeouts should be set from measured latency (e.g. just above the 8-second tail), not arbitrarily raised.
2. The team argues that because the failure rate is only 0.4%, it is not worth the engineering effort. How would you frame the impact and decide whether to invest?
Read the worked answer
0.4% sounds small but must be translated into business terms. If checkout volume is, say, 50,000 per day, 0.4% is 200 failed checkouts daily. Multiply by average order value and by the share of customers who abandon rather than retry successfully; even a conservative estimate usually dwarfs the engineering cost. Also count secondary effects: support tickets, refund handling, and reputational damage from bursts that coincide with peak traffic -- the worst possible time.
Framing for the decision:
- Quantify: failures per day, revenue at risk, support cost, and peak-hour concentration.
- Note that the failure is correlated, not independent: bursts hit many users at once, so the user-visible impact is worse than a uniform 0.4%.
- Note the coupling risk: because checkout saturation degrades product pages, the blast radius is larger than checkout alone, and it will get worse as traffic grows.
Decision approach: compare the cost of the fixes (backoff, circuit breaker, deadline propagation, pool isolation) against the quantified loss. These are well-understood, moderate-effort changes with low regression risk, so they usually clear the bar. If effort is constrained, prioritize the circuit breaker and pool isolation first -- they cap the blast radius -- then backoff and deadlines. Set a target (e.g. reduce burst-related failures by 90%) and monitor the failure rate and latency tail after rollout to confirm the investment paid off. Treating a correlated, growing failure as a rounding error is the mistake to avoid.