Artificial Intelligence Workloads and Performance
Artificial Intelligence tests how AI workloads use accelerated compute, memory, and interconnect. You need to reason about training versus inference, numerical precision, parallelism, and the metrics that reveal bottlenecks.
Core rules
- Memory is often the first constraint. Weights scale with parameter count and bytes per value; training adds gradients, optimizer states, and activations. Inference adds KV cache for autoregressive models.
- Mixed precision reduces memory and increases throughput, but accumulation and master weights may stay FP32. Check numerical stability before switching to FP8.
- Parallelism: data parallel replicates the model; tensor parallel shards layers within a node and is NVLink-friendly; pipeline parallel splits layers across nodes; ZeRO/FSDP shards optimizer state, gradients, and parameters.
- Scaling is limited by collectives. NCCL all-reduce, all-gather, and reduce-scatter traffic grows with model size and world size.
- Measure tokens/s, samples/s, time to first token, latency percentiles, GPU utilization, memory headroom, and interconnect throughput.
Worked example
A 70B-parameter model in FP16 needs 70B x 2 bytes = 140GB for weights. Training with FP16 gradients, 140GB, FP32 Adam moments, 70B x 8 bytes = 560GB, and FP32 master weights, 280GB, totals about 1.12TB before activations. Eight 80GB GPUs provide 640GB, so the job must shard optimizer state and parameters or use more GPUs; it does not fit by replication.
Traps
Equating parameter count with memory in bytes; forgetting KV cache and activations; assuming multi-GPU scaling is linear; and using tensor parallel across slow inter-node links.