Document-analyst throughput: TTFT collapses past 150 concurrent sessions

Utilization at 98% with flat tokens per second is the signature of wasted work, not of genuine compute saturation. Two causes compound here. First, 9,000 tokens of identical prefix are re-prefilled on every request, so 62% of GPU time is spent recomputing the same activations. Second, eviction every few seconds means cached blocks are thrown away and re-prefilled again - the eviction rate is a symptom of memory pressure caused by the prefill volume itself.

Ranked, no-new-hardware:

1. Prefix/radix caching, tenant-scoped, with the prefix versioned so a policy edit invalidates it. If 9,000 of ~12,000 input tokens are reused, prefill FLOPs fall by roughly three quarters - this is the largest single lever and requires no hardware.
2. KV-aware (cache-affinity) routing, so requests sharing a prefix land on the replica already holding those blocks, instead of any replica at random.
3. Chunked prefill to stop long prefills from blocking decode for interactive turns, protecting inter-token latency.
4. Autoscale on queue depth or TTFT-at-SLO goodput, not GPU utilization; utilization is the outcome, not a saturation signal, which is exactly why the scaler never fired.
5. Only if the above still misses: prefill/decode disaggregation across the existing 8 GPUs.

Doubling GPUs without fixing prefill reuse would roughly double capacity at double the cost - a last resort, not a fix.

Primary metrics: prefix cache hit rate (by tenant and by prefix version), p95 TTFT as a function of concurrency from 50 to 500, the prefill/decode split of GPU time, KV eviction rate, queue depth, and goodput measured as sessions served within the 3 s SLO.

Falsifiers. If cache hit rate is high and prefill time is low but TTFT still degrades past 150 sessions, the bottleneck is decode-side KV bandwidth or scheduling, not prefill - pivot to disaggregation. If eviction persists at high hit rate, the cache is being sized against the 32k max context rather than observed sequence lengths. If TTFT is fine but end-to-end task latency is poor, the problem has moved to the 2–6 tool calls and is no longer an inference-capacity question at all.

Method: a load generator replaying a realistic tenant mix with realistic prompt lengths, stepping concurrency while holding the query distribution fixed, run per replica group so one group can carry a candidate configuration. Report confidence intervals; a single 400-session run at 9 s is not enough to establish a new baseline.

Record the configuration (prefix version, cache size, routing policy, autoscaler signal) alongside each result, because these results are not comparable across config changes.

Three gates, all blocking.

Tenant isolation: cached blocks must be keyed by tenant and never reused across tenants, even with an identical prefix, or one customer's retrieved contract text can influence another's answer. Enforce this in the cache key and verify with a synthetic test that interleaves two tenants sharing a byte-identical system prompt and asserts no cross-contamination in outputs.

Invalidation: the policy prefix changes when legal edits it. Keying the cache on a content hash and pinning requests to the current version prevents a session from mixing an old cached policy prefix with new retrieved context - a silent, hard-to-detect failure. Test by rolling the prefix mid-load-test and confirming zero mixed-version responses.

Routing resilience: cache-affinity routing concentrates load and makes replica restarts expensive, since the affinity target disappears. Gate on measured TTFT recovery after a forced replica restart under load, and on rebalancing that prevents hot-spotting one replica.

Also cap any quality risk: run the retrieval-answer eval suite pre- and post-change on a fixed 300-question set, since caching changes nothing in theory but a mixed-version bug would show up as accuracy loss first.