Retries Turned a Slow Endpoint Into an Outage

The design guarantees a retry storm whenever upstream slows past the client timeout.

1. Client timeout (30 s) is shorter than server timeout (60 s) and shorter than normal p99 (45 s). So a meaningful share of requests was already timing out at the client during normal operation, while the server kept working on them. The system was running hot before the incident.
2. Two retries means up to 3x attempts per user action. When p99 moved to 70 s, almost every long request crossed 30 s, so nearly every request became three requests. That alone explains the tripled QPS - the users did not triple, the attempts did.
3. The retries land on an already-degraded upstream, consuming capacity on work whose result the client has already discarded. Error rate rises, which if the client also retries on error, compounds further.
4. Token spend matches the attempt count, not the request count. 3.4x spend for roughly the same user demand is the signature of duplicated generation, and the doubled posts confirm retried requests were not deduplicated - a non-idempotent POST was retried.

This is a positive feedback loop: slowness creates retries, retries create load, load creates slowness. No code change was needed to cause it; the timing relationships were the bug.

• Idempotency key per user action, generated client-side, stored server-side with the response. A retry with a known key returns the cached result or attaches to the in-flight generation instead of starting a second one. This alone removes the duplicate posts and most of the duplicate spend.
• Align timeouts with the layer below. If upstream may legitimately take 70 s, a 30 s client abort is a self-inflicted wound. Either make the client timeout exceed upstream p99 with margin, or - better - stream tokens so the client's liveness check is time-to-first-token, not time-to-completion. Then a slow answer is visibly progressing rather than ambiguously hung.
• Retry only what is safe to retry: 429 and 503, never on a timeout where the outcome is unknown, and never without an idempotency key. Exponential backoff with full jitter, capped attempts, and a retry budget (for example, retries may not exceed 10% of request volume) so retries cannot become the majority of traffic.
• Circuit breaker and bulkhead. When upstream error rate crosses a threshold, fail fast with a degraded response rather than adding load. Isolate long-generation traffic from any other calls to the same provider.
• Admission control. Queue at the edge with a bounded queue and shed load explicitly. A 429 from your own service is better than a 3x amplification of a provider incident.

Order matters: idempotency and streaming first, because they remove the reason for the retry; backoff tuning is cosmetic without them.

• Attempts per user action, as a distribution. Anything above 1.0 sustained is the leading indicator of amplification, and it would have shown the loop opening before QPS tripled.
• Retry ratio by reason (timeout vs 429 vs 5xx). A timeout-driven retry ratio is the dangerous one, because it means the server is doing work nobody will read.
• Tokens or cost per user action, not per request. This is the metric that catches duplicate generation; 3.4x is loud and unambiguous.
• Upstream p50/p95/p99 alongside your own client-abort rate. Alert on the ratio of client aborts to upstream completions; a persistent gap means the client is abandoning work the server is still paying for.
• Time to first token and tokens per second separately. Time-to-completion conflates queueing with generation and hides where the latency lives.
• Duplicate-output detector: count identical (user, prompt, output) tuples within a short window; post-hoc at minimum.

Also alert on the guardrail that was actually breached - upstream error rate - and make the alert reference the circuit-breaker state, so the person on call sees whether the system is currently shedding load or amplifying it. A dashboard is not a control; at least one of these should be an automatic trigger, not a chart.