Latency-Critical LLM Inference With an Idle GPU and a Saturated CPU
An LLM inference service runs on four A100 80 GB GPUs behind a Triton Inference Server. It serves three model variants, with a p99 latency SLO of 300 ms and a current throughput of 40 requests per second. GPU utilization is about 30%, but host CPU utilization is 90%, and the Python backend performs tokenization and prompt formatting before the model call. Dynamic batching is disabled because the team fears latency violations, so each request runs at batch size 1. The GPU memory is only 45% allocated. Users report intermittent p99 spikes to 700 ms, and the spikes correlate with traffic bursts rather than with model size. The business wants to double throughput without adding GPUs and without relaxing the SLO.
1. Where is the latency budget being spent, and what evidence would you collect before changing the serving configuration?
Read the worked answer
The correlation with traffic bursts and the 90% CPU utilization point to queueing and preprocessing, not GPU compute. At batch size 1 and 30% GPU utilization, the GPU is not the bottleneck; the host is. I would collect a request-level trace that separates queue wait, tokenization and prompt formatting, model forward pass, and response serialization. Triton exposes per-model and per-request metrics, and the client can add timestamps around the call. I would also measure CPU run-queue length, Python GIL contention, and the distribution of prompt lengths, because a few long prompts can occupy the Python backend and delay many short ones.
Check whether the 700 ms spikes are queueing delays or actual forward-pass times. If forward-pass time is stable at, say, 120 ms, then the extra 580 ms is outside the GPU. Also verify that the three model variants share the same GPU and whether model loading or memory pressure causes eviction. Evidence first prevents a blind switch to batching, which would add queue delay if the bottleneck is actually CPU serialization.
2. Recommend a serving architecture and batching policy. Justify whether MIG should be the first lever.
Read the worked answer
The first levers are moving preprocessing off the critical path and enabling bounded dynamic batching. Put tokenization and prompt formatting in a separate CPU worker pool with enough processes to avoid GIL serialization, or move them to GPU-accelerated preprocessing if the pipeline supports it. In Triton, enable dynamic batching with a small max_queue_delay, for example 5-10 ms, and preferred batch sizes that match the model's efficient range. A 10 ms queue delay is a small fraction of a 300 ms SLO and can raise GPU utilization substantially. Combine this with in-flight batching or a TensorRT-LLM backend if the model supports it, and consider quantization only after measuring accuracy impact.
MIG is not the first lever. MIG partitions the A100 into isolated slices and improves predictable latency and isolation, but it reduces the resources available to each model and does not fix a CPU-side bottleneck. With GPU utilization at 30%, partitioning would likely lower throughput and could increase per-request latency because each slice has less memory bandwidth. MIG becomes appropriate later if noisy-neighbor effects between the three variants persist after batching and preprocessing fixes, or if a strict per-tenant isolation requirement appears. Validate with a canary and compare p99, throughput, and GPU utilization.