Streaming Endpoint Latency Spikes Under Burst Load
You deploy a Claude-backed document-QA service on a managed API. Traffic is bursty: 10 req/min baseline, spiking to 400 req/min for ~90 seconds during batch uploads. The service streams responses to a web UI. At peak, users see 12-20 second time-to-first-token (TTFT) and some requests fail with 429s. You have a 60-second client timeout. p50 total response is 4s at baseline. Your provider account has a tokens-per-minute (TPM) and requests-per-minute (RPM) limit. Logs show retries are immediate (no backoff) and a retry storm follows each burst.
1. Diagnose the latency and 429 behavior, distinguishing provider-side limits from your own client behavior.
Read the worked answer
Two contributors, and the client is making it worse.
Provider-side: the burst (400 rpm vs 10 baseline) is a 40x spike. If your TPM/RPM limit sits below the burst's demand, the provider will throttle with 429s, and queued requests experience elevated TTFT as they wait. That's expected and not a bug - the question is whether you're managing it.
Client-side (the amplifier): immediate retries with no backoff turn a transient 429 into a retry storm. Each throttled request immediately re-fires, consuming more of the already-exhausted quota and pushing TTFT higher for everyone. The 60s client timeout then kills requests that might have succeeded, and those failures may themselves trigger retries. So the observed 12-20s TTFT and failures are partly self-inflicted.
Distinguish them by separating metrics: count 429s received vs. retries sent, and measure TTFT for first-attempt requests vs. retried ones. If first-attempt TTFT is fine and retried requests dominate the tail, the storm is the main problem. Also confirm actual usage against your TPM/RPM headroom - if the burst exceeds the limit even with perfect backoff, you need capacity or smoothing, not just better retries.
2. Propose concrete changes to survive the burst, and explain which ones reduce load versus which only hide latency.
Read the worked answer
Reduce load (real fixes):
- Exponential backoff with jitter on 429s, honoring any
Retry-After. This alone should collapse the retry storm. Cap retries (e.g., 3) and surface failures rather than looping. - Client-side rate limiting / token bucket sized to your TPM/RPM limit, so you never exceed quota and never trigger 429s in the first place. Queue excess and drain it.
- Admission control during bursts: shed or defer low-priority requests (e.g., non-interactive batch) so interactive QA keeps its quota.
- Request smoothing / debounce on the upload path so 400 rpm becomes a controlled ramp rather than a cliff.
- Raise limits or provision capacity if the burst is legitimate sustained demand - smoothing only helps if the work can wait.
Hide latency (UX only, doesn't reduce load):
- Streaming already helps perceived latency; ensure the UI renders tokens as they arrive so a 12s TTFT still feels responsive once it starts.
- Progress indicators and optimistic UI.
- These improve perception but if the underlying queue is saturated they just delay the bad news.
Timeout fix: a flat 60s client timeout is wrong for streaming - measure time-to-first-token and inter-token gaps, not total wall time, or long legitimate streams get killed. Set a TTFT timeout plus an idle-between-tokens timeout.
Prioritize the load-reducing changes; they address the cause. The latency-hiding changes are polish on top.