Performance Optimization
What this domain tests
This domain covers inference and training efficiency: latency versus throughput, batching, memory, numeric precision, and NVIDIA acceleration paths such as TensorRT and Triton Inference Server.
Core rules and concepts
- Latency is time per request; throughput is requests per second. Batching raises throughput but can raise per-request latency.
- The KV cache stores past keys and values during autoregressive decoding. It trades memory for speed and grows with sequence length, batch size, layers, and head dimension.
- Lower precision (FP16, BF16, FP8, INT8) reduces memory and can raise throughput; validate quality after quantization.
- Roofline reasoning: kernels are limited by memory bandwidth or compute. Long-context attention is frequently bandwidth-bound.
- FlashAttention reduces memory traffic by tiling attention; it changes the implementation, not the underlying math.
Worked example
A 7B-parameter model in FP16 needs 7e9 x 2 bytes = 14 GB for weights. Add a KV cache for batch 8, 4,096 tokens, 32 layers, 32 heads, head dimension 128: per token per layer, 2 x 32 x 128 x 2 bytes = 16 KiB; times 4,096 tokens = 64 MiB; times 32 layers = 2 GiB per sequence; times 8 sequences = 16 GiB. Total is roughly 30 GB, which exceeds a 24 GB GPU. The fix is to shrink batch or context, or use FP8 weights and a paged KV cache.
Traps that cost marks
- Optimizing mean latency while p99 latency degrades.
- Forgetting KV cache memory in capacity planning.
- Assuming quantization is lossless.
- Benchmarking at batch size 1 and claiming throughput results.
- Attributing a speedup to a kernel when the input shape simply changed.