Cutting p95 Latency and Cost on a Mixed-Traffic LLM Application

Order by impact per unit of quality risk, and fix the measurement gap first.

1. Stop the retry amplification. Timeouts causing retries is a feedback loop: load raises p95, which causes more timeouts. Add a retry budget and idempotency, and set client timeouts above server p99. This is nearly free and reduces both cost and load - do it before anything else, or every other optimization is measured against a moving baseline.
2. Cache the FAQ traffic. 70% of requests with near-identical phrasing is the largest single lever on both cost and latency. Semantic or normalized-prompt caching, with a TTL and an invalidation path on source-document changes.
3. Trim the system prompt. 4,000 tokens per call across 500k calls is a large fixed cost and a fixed latency contribution; most such prompts contain examples that can be moved to retrieval or removed.
4. Model routing. Send the easy 70% to a smaller model, keep the hard 20-30% on the frontier model.
5. Fine-tuning/distillation last - highest effort, and only justifiable once routing shows which traffic the small model actually fails.

Measure quality at each step, because steps 2-4 change answers.

Two caches, different risk profiles.

• Exact-match cache on normalized prompt (case, whitespace, punctuation): very low risk, cheap, and may capture less of the 70% than expected if phrasing varies.
• Semantic cache with an embedding similarity threshold: captures paraphrases but introduces the real risk - a near-identical question with a materially different answer (different product, different region, different policy year) gets served the wrong cached answer, confidently. Threshold tuning trades hit rate against wrong answers, and the failures are silent.

Mitigations: scope cache keys by tenant/region/product where the domain demands it; never cache answers that depend on time-sensitive data; store the source answer's citations and re-validate on serving; log cache hits so you can sample and audit them.

Invalidation on document change is the operational hazard - a stale cached answer survives a policy update. Key the cache to a content version so updates invalidate cleanly.

Judgement call: start exact-match, measure hit rate, then widen to semantic only with an eval set of 'similar question, different correct answer' pairs to prove the threshold is safe.

Routing is a classification problem, and the classifier's errors are asymmetric: sending a hard question to the small model degrades quality visibly; sending an easy one to the large model only costs money.

• Bias the router toward the large model. Tune for high recall on 'this needs the big model' rather than balanced accuracy.
• Bootstrap the router from existing data rather than guessing: run a sample of real traffic through both models, score with the calibrated judge plus spot-checked humans, and derive the decision boundary from where they diverge.
• Make failure detectable at runtime: low retrieval confidence, the small model's own uncertainty signal, or answers below a length/format threshold escalate to the large model. A two-stage cascade costs one small call on the easy path and two calls only on hard ones - usually cheaper than routing on a classifier alone.
• Canary the router and keep the quality gate attached, since routing is a model change in disguise.

Explicitly state the bar: the small model must match the large model on the FAQ slice within the judge's noise, on a set of at least a few hundred evaluated questions. Otherwise the '70% easy' assumption is unverified and you are trading quality for cost on faith.