AI Software Stack and Workload Operations
AI Software Stack and Workload Operations
This domain tests the container-to-cluster software path: NVIDIA driver, CUDA toolkit, cuDNN/NCCL, NGC containers, Kubernetes device plugin, Triton Inference Server, and job observability.
Core rules and concepts
- The host driver must be compatible with the container's CUDA runtime; the NVIDIA Container Toolkit exposes GPUs to containers.
- Pin image digests, not latest tags, for reproducible training and inference.
- Use NCCL environment variables and debug logging to diagnose multi-GPU hangs.
- In Kubernetes, request nvidia.com/gpu or a MIG resource; the device plugin advertises capacity.
- Monitor GPU utilization, memory, ECC, and job-level throughput; container CPU starvation can look like GPU idle.
Worked example
A PyTorch job slows after a container update. nvidia-smi shows healthy GPUs, but NCCL logs show it falling back to PCIe and a socket interface. The fix is to run with host networking or the correct RDMA interface and confirm nvidia-smi topo -m shows NVLink. The reasoning: the new image changed NCCL transport defaults, so collectives crossed the wrong fabric. Verify the fix with a cluster-wide all-reduce before reopening the job.
Traps that cost marks
Assuming a newer CUDA image always works with an older driver; forgetting --gpus or the device plugin; using latest tags; and blaming the GPU when the DataLoader is CPU-bound.