Retry Amplification Takes Down a Payments Platform
A payments platform runs three services - checkout, inventory and a ledger - behind one load balancer. Checkout calls inventory with a 200 ms timeout, up to three retries with exponential backoff from a 100 ms base and no jitter, and no circuit breaker; the HTTP client library also retries once on connection failure. On a Saturday the inventory database failed over. For four minutes inventory's p99 rose to 900 ms while it still served roughly 60% of requests correctly. Checkout receives 400 rps and handles each request on a thread from a shared 200-thread pool that serves all of its endpoints. Within 30 seconds the whole platform was unavailable: checkout's pool was exhausted, the ledger - which shares no backend with inventory - began timing out, and the on-call engineer restarted checkout twice, which lengthened recovery. The post-incident proposal is to raise the timeout to 2 s and add a fourth retry.
1. Quantify what the retry policy did, and explain why the ledger degraded when it shares nothing with inventory.
Read the worked answer
Step 1 - attempts per request. A request that fails at the transport layer can consume one connection retry plus up to three policy retries, so the worst case is four attempts, each carrying a 200 ms timeout plus backoff delays of roughly 100, 200 and 400 ms. Worst-case occupancy per request is therefore about 0.8 s of timeouts plus 0.7 s of backoff, close to 1.5 s.
Step 2 - Little's law. At 400 rps and 1.5 s of occupancy, the concurrency demanded is 400 × 1.5 = 600 in-flight requests against a 200-thread pool. The pool saturates at about 133 rps of failing traffic, and every request that cannot get a thread queues. That is the amplification: a service that is still succeeding on 60% of calls is taken out of service by its caller.
Step 3 - why 200 ms is the wrong timeout. Inventory's p99 is 900 ms during the incident. A 200 ms timeout abandons work that would have completed, then re-sends it, so the retries are pure added load on a database that is already recovering. Each retried write also risks double execution if the reserve call is not idempotent.
Step 4 - why the ledger fell over. The shared 200-thread pool means inventory latency consumes threads that ledger calls also need. This is a bulkhead failure, not a ledger failure. The same load balancer keeps accepting connections, so clients see an apparently total outage.
Step 5 - why restarts hurt. Restarting checkout killed in-flight requests, produced a reconnect and retry storm from clients, discarded warm JIT compilation and caches, and required re-establishment of hundreds of connections - all while the dependency was still degraded. Restarts treat a saturation symptom as a crash.
2. Redesign the resilience pattern. State the concrete values you would set and why each one follows from the numbers.
Read the worked answer
Timeouts. Set the per-attempt timeout above the healthy p99.9 (say 300–400 ms) and set an overall deadline per user request (about 1 s) that caps the total across attempts. A timeout shorter than the dependency's degraded p99 is not protection, it is extra load.
Retries. Two attempts maximum for inventory, full jitter (sleep = random(0, base × 2^attempt)), and no transport-layer retry stacked on top of the policy retry - the double retry is the multiplier nobody budgeted for. Only retry idempotent operations; the reserve call must carry an idempotency key so a duplicate attempt is a no-op.
Retry budget. Cap retries at roughly 10% of successful calls per client (a token bucket). This is the single most effective control, because it makes amplification impossible during a partial outage: as the success rate falls, the budget to retry falls with it.
Circuit breaker. Trip on error rate or on latency above the healthy p99.9 over a short window, open for a few seconds, then half-open with a single probe. When open, fail fast and return a degraded response instead of holding threads.
Bulkhead. Give each dependency its own connection pool and its own concurrency limit with a semaphore (for example 80 concurrent inventory calls). Threads waiting on inventory can no longer starve ledger traffic. This directly addresses the fact that the ledger has no shared backend yet still fell over.
Load shedding. Inventory should reject excess work at ingress rather than queue it, and checkout should prefer failing a fraction of requests quickly over failing all of them slowly.
Operational rule: during an incident, do not restart a saturated service; shed load, disable the retry path, and let the dependency recover.
3. How would you validate this design before the next incident, and which single metric would have caught this early?
Read the worked answer
Validation must run at production rate against an injected fault. A unit test with a mocked slow dependency proves nothing, because the failure mode is concurrency and queueing.
- Fault injection in staging. Use a proxy in front of inventory to add 900 ms latency to 40% of calls, at 400 rps of closed-loop or, better, open-loop traffic for at least ten minutes. Assert three things: checkout's thread pool occupancy stays below 70%, ledger p99 stays within SLO, and total requests to inventory stay within 1.2× of the offered rate. The last assertion is the definition of 'no amplification' and should be an automated gate in the pipeline.
- Chaos game day. Repeat with the database failover, a DNS failure and a partial packet loss, and rehearse the runbook: the correct action must be a load-shed or breaker-open, not a restart. Time the recovery to a healthy state.
- Load test the breaker itself. Verify that it opens, that half-open probes are rate-limited, and that it closes without a stampede.
The single metric: the retry amplification ratio, meaning attempts sent to inventory divided by logical requests made. It is normally 1.0 and it rises before latency collapses. Alert when it exceeds about 1.3 for a minute. It is better than error rate because error rate lags - checkout still reported success on many requests while it was busy flooding inventory.
Supporting metrics worth dashboards: per-dependency concurrency and pool saturation, requests-in-flight versus thread count, circuit-breaker state changes, and an error-budget burn rate. Together they turn the next incident into a capacity decision rather than a scramble.