The 12-Session Ceiling on an 80 GB A100
A team deploys an 8B chat model on a single 80 GB A100 behind an OpenAI-compatible server. Architecture: 32 layers, 32 attention heads, head dimension 128, fp16 weights and fp16 KV cache. They configure max_model_len = 8192 and max_num_seqs = 16, reasoning that an 8B model in fp16 is only 16 GB, so there is ample headroom. In load testing, 12 concurrent sessions of 8,000 tokens run cleanly. The 13th concurrent session triggers a CUDA out-of-memory error at startup of its prefill. In the seconds before the crash, inter-token latency for the 12 healthy sessions roughly doubles. The serving framework reserves KV blocks up to max_model_len x max_num_seqs and runs with gpu_memory_utilization = 0.85.
1. Compute the KV cache footprint and use it to explain both the 12-session ceiling and the latency doubling that precedes the crash.
Read the worked answer
Per-token KV size. 2 tensors (K and V) x 32 layers x 32 heads x 128 dims = 262,144 elements. At 2 bytes each that is 524,288 bytes = 512 KiB per token.
Per sequence at full context. 512 KiB x 8,192 tokens = 4 GiB. This is the number that was missing from the team's reasoning: context length, not parameter count, sets the footprint.
Budget. 80 GB x 0.85 = 68 GB usable. Weights are 16 GB. CUDA context, activation buffers and graph captures take roughly 4 GB. That leaves about 48 GiB for KV blocks, which is exactly 12 sequences (12 x 4 GiB = 48 GiB). The 13th needs another 4 GiB and fails. So the ceiling is arithmetic, not a bug.
Why latency doubles before the crash. Once the block pool is full, the scheduler cannot allocate blocks for newly generated tokens. Paged-attention servers handle this by preempting running sequences and recomputing them later from their prompt. That recompute work is charged to the same GPU, so every still-healthy sequence sees its inter-token latency climb before the process dies. The latency spike is the preemption signal, not an unrelated performance regression.
2. The team needs at least 8x the current concurrency on this hardware. Rank the available mitigations by risk and recommend a combination.
Read the worked answer
Options, cheapest first:
- fp8 KV cache - halves per-sequence cost to 2 GiB, doubling capacity to ~24 sequences. Small, measurable quality risk; validate on the eval suite before committing. Usually the best ratio of effort to gain.
- Reduce max_model_len to 4096 - same 2x gain, but it truncates real traffic. Check the actual prompt+output length distribution first; if p99 is 3,000 tokens, 4,096 is nearly free, if p99 is 7,000 it silently degrades quality.
- Raise gpu_memory_utilization to 0.95 - buys ~8 GB, i.e. two more sequences, at the cost of fragmentation OOMs under load. Fragile; not a fix on its own.
- Cap max_num_seqs at 12 and queue the rest - makes the failure mode graceful (queueing instead of crash) but does not add throughput.
- Tensor parallelism across 2 GPUs - weights and KV heads shard, so per-GPU KV drops to 2 GiB per sequence and capacity roughly quadruples. Correct answer if the load is real, but doubles cost.
Recommendation: fp8 KV cache plus an explicit max_num_seqs matched to the measured pool, plus admission control so overflow queues rather than OOMs. Do not treat items 2 and 5 as interchangeable - one trades quality, the other trades money. Re-run the load test at the target concurrency with worst-case-length prompts, because average-length tests understate prefill peaks.