Diagnosing Low GPU Utilization in an Inference Service
A team runs a real-time image classification service on 4 GPUs. Each GPU hosts one model instance. Traffic averages 900 requests per second with bursts to 2,500. Median latency is 40 ms but p99 is 380 ms, and the SLO is 150 ms at p99. Monitoring shows GPU utilization at 35%, GPU memory at 88%, and host CPU at 95%. The model is a fixed-shape convolutional network using batch size 1 per request. The team's first instinct is to add two more GPUs. Before approving the spend, the platform lead asks for a diagnosis of why utilization is low while latency is high, and whether more GPUs would help.
1. Given 35% GPU utilization and 95% host CPU, what is the most likely bottleneck, and how would you confirm it?
Read the worked answer
The combination of low GPU utilization and saturated host CPU points to a host-side bottleneck, not a GPU compute limit. When the GPU is only 35% busy while the CPU is at 95%, the GPU is waiting for work to arrive. Common causes include request preprocessing, tokenization or image decode on the host, Python-level overhead in the serving loop, per-request synchronization, and single-request batching that prevents the GPU from amortizing launch overhead.
Confirmation steps:
Profile the request path end to end. Measure time spent in image decode, preprocessing, model invocation, and postprocessing. If decode and preprocessing dominate, the GPU is starved by the CPU.
Measure GPU kernel time versus wall-clock time per request. A large gap between kernel execution time and total request latency means the GPU is idle waiting.
Check for synchronization points. Per-request calls that force a device sync serialize the pipeline and prevent overlap.
Inspect the serving framework's batching configuration. Batch size 1 per request means the GPU processes tiny workloads with high launch overhead relative to useful work.
Correlate host CPU with request rate. If CPU saturates before GPU utilization rises, the host is the constraint.
Memory at 88% is a separate concern: it limits how much batching headroom exists. Even if the host bottleneck is fixed, memory may cap the batch size, so the team should plan to reduce per-instance memory footprint or use a smaller precision where accuracy allows.
2. Would adding two GPUs fix the p99 latency problem? Justify your answer and propose an alternative plan.
Read the worked answer
Adding GPUs would likely not fix p99 latency, and could make it worse in the short term. The GPU is 35% utilized, so compute capacity is not the constraint. More GPUs spread across the same saturated host CPU and the same request-routing layer would add more contention on the host and the network path. The p99 spikes at 380 ms are consistent with queueing during bursts: at 2,500 requests per second, a host-side pipeline that is already at 95% CPU will queue requests, and queueing delay grows nonlinearly as utilization approaches saturation.
Alternative plan, in priority order:
Move preprocessing off the critical path. Use hardware-accelerated decode, precompute where possible, and parallelize preprocessing across CPU workers sized to the host core count.
Introduce dynamic batching. Group requests into batches sized to fit within the 88% memory headroom. Batching raises GPU utilization and amortizes launch overhead, which typically improves both throughput and tail latency.
Reduce per-request synchronization so the GPU pipeline overlaps with host work.
Re-measure. If GPU utilization rises above roughly 70% and p99 still exceeds 150 ms, then compute is genuinely constrained and adding GPUs is justified.
Only after the host bottleneck is resolved should the team scale horizontally, and then it should scale based on measured utilization per instance.
The key judgement: low GPU utilization plus high CPU utilization is a signal to fix the pipeline, not to buy hardware.