Sizing a 13B Inference Fleet When Memory Is Not the Constraint
You are sizing a serving fleet for a 13B chat model at FP16 (26 GB of weights) on 4-GPU A100 80 GB nodes. Workload: 1,500 input tokens, 200 output tokens, 20 requests/s at peak. SLO is 3.0 s p95 end-to-end. From the profiling run on target hardware: prefill of 1,500 tokens takes 180 ms; per-token decode latency is 11 ms at batch 1, 14 ms at batch 4, 18 ms at batch 8, and 26 ms at batch 16; KV cache is 0.8 MB per token per sequence. The ops team wants to run nodes at no more than 60% of their latency-limited capacity to absorb bursts and node loss. Someone proposes buying 80 GB cards instead of 40 GB cards to 'fit more concurrent requests'.
1. Derive the largest batch size that meets the SLO, the resulting per-GPU throughput, and the number of GPUs required. Show each step.
Read the worked answer
1. Budget per decoded token. End-to-end budget 3,000 ms minus 180 ms prefill = 2,820 ms for 200 tokens, so 14.1 ms/token.
2. Largest batch that fits. 11 ms (batch 1) and 14 ms (batch 4) are inside 14.1 ms; 18 ms (batch 8) is not. Max batch = 4. Note the trap: batch 8 roughly doubles throughput per GPU but violates the SLO, and an SLO is not negotiable with throughput.
3. Per-GPU throughput. 4 sequences / 0.014 s per token = 286 tokens/s = 1.43 requests/s (at 200 output tokens each).
4. Fleet size. 20 req/s / 1.43 = 14 GPUs at 100% of the latency limit. At 60% target utilization: 14 / 0.6 = 23.3 -> 24 GPUs = 6 nodes of 4.
Sanity check the cost: 24 GPUs is a large fleet for a 13B model, and it is entirely driven by the per-token decode latency at small batch - not by memory, not by model size.
2. Does the 'buy 80 GB instead of 40 GB' proposal help? Check the memory math and the reasoning.
Read the worked answer
Memory is nowhere near binding at the chosen operating point.
KV at batch 4: 4 x (1,500 + 200) x 0.8 MB = 5.4 GB. Plus 26 GB of weights plus workspace: roughly 33-35 GB. That fits on a 40 GB card with room to spare, and on an 80 GB card it uses less than half.
The proposal confuses two different constraints. Bigger cards raise the memory-limited batch cap. This deployment is latency-limited: batch 8 already misses the 3.0 s SLO, so the memory headroom that an 80 GB card unlocks would produce batches that cannot be served within the SLO. Buying capacity you cannot use is pure capex.
What would genuinely help, in order:
- Lower per-token latency, not more memory: FP8/INT8 weights, speculative decoding, or a smaller/distilled model - anything that moves the 14 ms/token curve.
- Cap output length. The 200-token assumption is the dominant term; a 120-token cap with a fallback to 'expand' cuts decode time ~40%.
- More replicas of the cheap card if the latency curve is unchanged; 6 nodes of 4 is the same whether each card is 40 or 80 GB.
Only revisit 80 GB cards if the workload shifts to long contexts where KV, not decode latency, sets the batch cap.
3. Traffic is forecast to double next quarter. Rank the interventions and identify which one you would pilot first, given the constraint that the SLO cannot be relaxed.
Read the worked answer
Doubling to 40 req/s means ~47 GPUs at 60% utilization on the current curve - the fleet cost balloons far faster than the model does. That reframes the problem: fix the per-token cost before buying more cards.
Ranked by leverage against the latency constraint:
- Reduce output tokens. Directly proportional to decode time and requires no new hardware. Highest certainty, lowest risk. Pilot first.
- Speculative decoding / a smaller draft model. Attacks the 14 ms/token curve itself, which is the actual bottleneck. Keep the acceptance rate monitored; a bad draft model adds latency instead of removing it.
- Quantization (FP8/INT8). Cuts weight memory and usually per-token latency, but must be validated on the eval suite - a faster model that fails the quality bar is not an option.
- More replicas. Always works, scales roughly linearly, costs linearly. This is the fallback, not the plan.
- Bigger GPUs. Last, and only for KV-bound workloads.
Pilot order: output cap (a config change, measurable in days) alongside speculative decoding on a canary slice, with the SLO and the quality eval as gates. Only after the new latency curve is known do you re-run the fleet arithmetic - the 24-GPU number is an artifact of today's curve, not a constant. And keep the 60% headroom: it is what absorbs a node loss during a burst, so do not quietly spend it to hit a launch date.