Peak-hour timeouts on a two-node dedicated deployment

Both, and the arithmetic shows why neither fix alone works.

Capacity: 180 requests/min is 3 requests/s. At up to 800 output tokens each, peak output demand is about 2,400 tokens/s against roughly 2,200 tokens/s of aggregate capacity across two nodes. Demand exceeds supply, so requests queue. That explains p50 rising to 6s - even the median request is waiting - and the tail at 41s.

Configuration amplifies it. A fixed 800-token cap ignores the fact that most support answers are far shorter; generating to the cap wastes the scarcest resource. 2,500 tokens of context per request adds prefill load and latency without evidence that all of it is used. The 30s gateway timeout is shorter than the observed p95, so slow-but-successful requests are converted into errors. Client retries without a budget then multiply load roughly 3×, pushing the system further into saturation - a classic retry storm where the failure mode feeds itself.

So: provision more capacity or reduce demand per request, then fix timeout and retry behavior. Order matters. If you raise the timeout first, you convert timeouts into very slow successes and hold connections longer. Reduce token demand and add admission control first, then scale, then tune timeouts against measured latency.

Reduce demand per request:

• Set max_new_tokens per task type (for example 250 for FAQ answers, 500 for troubleshooting) and stop on end-of-sequence rather than fixed length.
• Trim retrieved context: rerank and keep the top 5–8 passages instead of everything, and cap context at a measured budget. Verify answer quality does not drop.
• Enable streaming so time-to-first-token drives perceived latency; users see progress while generation continues.

Control queueing:

• Put an admission-control limit in front of the endpoint with a bounded queue and a fast, explicit busy response instead of a 30s hang. Shedding load early protects the requests already in flight.
• Separate endpoints or deployment sizes for heavy and light requests so a long summarization job cannot block short chat turns.
• Pre-scale on a schedule ahead of the Monday 09:00 peak rather than waiting for reactive autoscaling, which lags by minutes.

Fix client behavior:

• Exponential backoff with jitter, a retry budget (for example, at most 10% of traffic may be retried), and idempotency keys so a retried request does not duplicate side effects.
• Set the gateway timeout above measured p99 for accepted requests, and make the client's timeout consistent with it.

Re-measure: target p95 under 8s at 180 requests/min with retries disabled, then re-enable retries and confirm stability.

Replace the in-place, both-nodes-at-once rollout with a blue/green or canary pattern.

• Maintain two deployment revisions. Send a small canary slice (5%, then 25%) to the new revision while the old one serves the rest. Compare p95 latency, error rate, timeout rate, and a sampled answer-quality score against the incumbent before increasing traffic.
• Gate promotion on automated checks plus a manual approval for prompt or model changes that affect answer content. Roll back by shifting traffic, not by redeploying.
• Drain connections from the old revision before terminating it; in-flight generation requests should complete or be retried with backoff, not cut off.

Because there is a scheduled peak, freeze releases on Monday mornings and during month-end. Run canaries at low-traffic windows.

Instrument the rollout: per-revision latency percentiles, time-to-first-token, tokens per second, queue depth, and error budget consumption. Track the SLO monthly and review the error budget; if releases consume more than a set share, slow the release cadence.

Finally, record each release as an artifact - model version, prompt version, retrieval configuration, evaluation results - so a regression can be traced to a specific change and the factsheet stays accurate.