TensorRT-LLM Serving for Long-Prompt RAG
A team serves a 70B parameter model on 8×H100 80GB GPUs using TensorRT-LLM, tensor parallelism 8, and FP8 weights. The workload is RAG: 1,800 input tokens, including a shared 1,200-token system and instruction prefix, and 400 output tokens, at concurrency 32. The SLOs are p99 time-to-first-token under 800 ms and p99 time-per-output-token under 60 ms. Observed p99 TTFT is 2.6 s, mean TPOT is 44 ms, p99 TPOT is 71 ms, and throughput is 1.2 requests per second. KV cache occupancy is 86%, with about 8 requests evicted per minute. The scheduler caps max_batch_size at 8, and max_seq_len is fixed at 8,192 even though p95 context length is 2,300 tokens. GPU utilization swings from 95% during prefill to 40% during decode. The team cannot add GPUs this quarter.
1. Identify the dominant bottleneck and explain the TTFT and TPOT symptoms.
Read the worked answer
TTFT at 2.6 s with concurrency 32 and max_batch_size 8 indicates queueing plus repeated prefill. Requests wait for a batch slot, and every request recomputes the same 1,200-token prefix. The 95% prefill to 40% decode utilization swing shows there is no effective in-flight batching or overlap; decode runs in small batches and is periodically interrupted by prefill. That explains p99 TPOT at 71 ms: decode tokens are delayed by long prefill chunks and by a batch that is too small to amortize weight reads.
KV cache pressure compounds the problem. Occupancy at 86% with evictions means the allocator is reserving for an 8,192-token maximum even though p95 context is 2,300. Evicted requests must recompute, creating TTFT spikes. Measure TTFT at concurrency 1 versus 32 and record prefill time per request. If single-request TTFT is under 800 ms, queueing and scheduling dominate. If it is also high, prefill compute or memory bandwidth is the bottleneck.
2. Propose a TensorRT-LLM configuration plan and expected impact.
Read the worked answer
Enable in-flight batching and chunked prefill with a chunk size of 512-1,024 so decode can interleave with prefill and p99 TPOT improves. Enable prefix caching or KV cache reuse for the shared 1,200-token prefix, which removes roughly two-thirds of the prefill work for most requests. Replace the fixed 8,192 max_seq_len with 4,096 and a fallback path for longer contexts, and use paged KV cache to reduce fragmentation and evictions. Raise max_batch_size and max_num_tokens within memory limits; the current cap of 8 is leaving decode underutilized.
Expected effects: prefix caching cuts TTFT and prefill load; chunked prefill lowers p99 TPOT; larger decode batches raise throughput. Risks include prefix-cache invalidation, chunk sizes that are too large and still hurt TPOT, and truncation if max_seq_len is reduced carelessly. Tune chunk size and batch token budget with a sweep. Do not simply increase max_seq_len or batch size without checking KV memory, or evictions will worsen.
3. Design a load test and rollout validation plan.
Read the worked answer
Replay production arrival rates rather than a closed loop, and sweep concurrency from 1 to 64. Use the real input-length distribution, including p99 contexts, shared-prefix hit rates, and long-tail requests. Measure TTFT and TPOT at p50, p95, and p99, goodput under the SLOs, KV occupancy, eviction rate, prefix-cache hit rate, GPU memory, and error rate. Compare against the current configuration at the same offered load.
Success criteria: p99 TTFT under 800 ms and p99 TPOT under 60 ms at concurrency 32, no OOM or eviction storm, and error rate below 0.1%. Canary 10% of traffic with automatic fallback. Monitor prefix-cache effectiveness and whether long-context requests are truncated. If the SLOs hold only at low concurrency, the remaining fix is capacity or model parallelism, not scheduler tuning. Document the risk that a workload shift toward longer contexts will invalidate the 4,096-token configuration.