Systems Operations, Monitoring, and Troubleshooting
Health and telemetry
Operational tooling centers on nvidia-smi (utilization, memory, temperature, power, ECC errors) and DCGM (Data Center GPU Manager) for fleet-wide metrics, health checks, and diagnostics. Questions often ask which tool fits: nvidia-smi for quick local inspection, DCGM for monitoring at scale and policy enforcement.
Common failure modes
Recognize symptoms: out-of-memory errors (batch too large or leak), GPU falling off the bus (Xid errors in the system log), thermal throttling (clocks drop under sustained load), and ECC errors (correctable vs uncorrectable). A trap is treating every slowdown as a code problem when it is throttling or a degraded link.
Xid and logs
Xid messages in kernel logs indicate driver or hardware faults. Candidates should know to check system logs and DCGM diagnostics before replacing hardware. Another trap: ignoring that a container may hide the host's GPU error state.
Capacity and scheduling
Understand MIG for isolation, time-slicing for sharing, and Kubernetes device plugins plus GPU operator for orchestration. Know that MIG instances appear as separate devices to schedulers.
Worked example
A training job slows after hours of running. Check nvidia-smi for clock throttling and temperature first; sustained thermal or power limits explain gradual slowdown better than a sudden code regression. Rule of thumb: when performance degrades over time rather than at start, suspect thermals, power caps, or memory fragmentation before the model.