The 9-Second p95: A Support Copilot That Misses Its SLA

Two different problems are hiding in one p95 number.

1. Cold start (the 20-second morning request). Scale-to-zero means the first request after idle pays for capacity acquisition and model load. This is not a prompt-size problem and no amount of context trimming fixes it.
2. Prefill cost on the tail (the 9.4s p95). ~6,000 input tokens must be processed before the first output token, and prefill cost grows with input length. Long conversations plus 8 chunks put the worst requests in the slow lane. The Vector Search call for those chunks also carries its own tail, and if retrieval is sequential with generation rather than overlapped, the tails add.

Ranked fixes:

• Warm capacity during support hours. Set a minimum replica count or scheduled warm-up for business hours, and let it scale to zero overnight. Cost is real but bounded and predictable - accept it, because the SLA is the product.
• Cut input length. Keep the last 3–4 turns verbatim plus a rolling summary of older turns; reduce retrieved chunks from 8 to 4–5 with a reranker on top of a larger candidate set. This attacks prefill time and cost simultaneously.
• Stream tokens. Doesn't change total latency but changes perceived latency materially; cheap to adopt.
• Measure the retrieval leg separately before assuming generation is the only tail.

Tradeoff I accept: fewer chunks can reduce recall. I would not cut chunks until I have recall@k on the evaluation set proving the reranked 4-chunk configuration holds quality. Trimming context without that measurement is guessing.

Input tokens are 60% of the bill, so the levers are the context, not the output.

Pull first, in this order:

1. Context reduction (same levers as the latency fix - conversation summarization, reranking down to 4 chunks, deduplication of near-identical chunks). This is the only lever that improves latency and cost at once, which makes it the default first move.
2. Tiered routing. A small model handles intent classification, FAQ-shaped questions, and clarification turns; only genuinely hard queries reach the 70B. Route on a classifier you can evaluate, not on keywords.
3. Answer reuse for the true FAQ tail. A high-frequency question set served from a curated store with a semantic lookup fallback to the full pipeline.
4. Provisioned throughput once the post-optimization volume is stable and predictable - the unit economics improve above a sustained utilization threshold.

Refuse to cut: evaluation, retrieval quality, and the redaction/logging path. A cost program that quietly disables inference tables or the offline eval harness buys this quarter and loses the next one - you lose the ability to know whether the cheaper system is still good. Also refuse to swap to a smaller model on cost grounds alone: model changes go through the evaluation gate, or they aren't changes, they're bets.

Provisioned throughput is a volume-and-variance decision. Move when: (a) post-optimization traffic is sustained and reasonably flat within each shift, (b) the SLO genuinely requires guaranteed capacity and warm models, and (c) the committed spend is below the projected pay-per-token cost at that utilization. The failure mode to avoid is provisioning for peak and paying for idle troughs - if the diurnal swing is as extreme as 40 rpm to near zero, a hybrid (provisioned for business hours, pay-per-token overnight) often wins.

A smaller fine-tuned model is a task-fit decision, not a cost decision. It is justified when:

• The failure analysis shows the residual errors are style, format, or consistency problems (citation format, refusal phrasing, structured output), not knowledge problems - fine-tuning teaches behavior, not facts that change.
• A distillation set built from the 70B's outputs on your traffic exists and is clean.
• The small model, evaluated on the same golden set, is at parity on the scorers that matter and degrades gracefully on the slices where it loses.

Explicitly do not fine-tune to fix knowledge staleness. A model fine-tuned on last quarter's policies will answer confidently and wrongly, with no retrieval trace to blame. Staleness is a corpus and indexing problem. Also weigh operations: a fine-tuned model is another artifact to version, re-tune, evaluate, and serve, so it needs an owner.