AI Operations
What this domain actually tests
AI Operations covers running GPU clusters reliably: monitoring, scheduling, health validation, and recovering from failures without losing days of training.
Core concepts to own
Monitoring centers on NVIDIA DCGM and DCGM Exporter feeding Prometheus and Grafana. Know the distinction between GPU utilization (SM activity), memory utilization, power draw, temperature, ECC errors, and XID error codes. Health validation uses DCGM diagnostics at increasing levels (from a quick smoke test to full stress and memory tests) and NCCL all-reduce bandwidth tests to confirm the fabric.
Scheduling means Slurm or Kubernetes with the NVIDIA GPU Operator and MIG device plugin, plus topology-aware placement so communicating ranks land on the same NVLink domain. Operations also include firmware, driver, and CUDA compatibility management through the support matrix, node draining and cordoning, and checkpointing strategy aligned to expected mean time between failures.
Traps
GPU utilization near 100% does not prove useful work - a stalled collective can look busy. Prometheus counters and gauges are confused. XID errors are treated as generic failures when specific codes point to specific faults (memory, fabric, driver). Forgetting to drain a node before maintenance is a classic operational error.
Worked example
A 512-GPU job loses a node every 4 hours. With a 30-minute checkpoint interval, expected lost work is about half the interval, roughly 15 minutes per 240 minutes - about 6% overhead. Hourly checkpoints would waste roughly 12.5%; every 5 minutes would waste under 1% but add I/O cost, so interval selection is a tradeoff, not a constant.