I85PREP
⌘ K
NVIDIA NCP-AII study materials
NVIDIA NCP-AII study resources

NVIDIA NCP-AII: Monitoring, Validation, and Troubleshooting

NVIDIA NCP-AII · Work through your exam, one topic at a time.

All guide sections

Monitoring, Validation, and Troubleshooting

What this domain tests

Separating a real hardware fault from a configuration problem, quickly and with evidence. Expect DCGM diagnostics, Xid interpretation, ECC and retired-page inspection, log collection, and burn-in methodology.

Core rules

  • DCGM is the first tool. dcgmi discovery -l lists GPUs, dcgmi diag -r 1 is a fast sanity pass, and dcgmi diag -r 3 runs a sustained stress test suitable for acceptance. Never run diagnostics against a GPU with active workloads.
  • Xid codes are the vocabulary. Xid 79 means the GPU fell off the PCIe bus - check riser seating, power, and slot errors before RMA. Xid 48 is an uncorrectable ECC error. Xid 63/64 indicate ECC page retirement events, and Xid 13/31 usually come from an application's illegal memory access rather than failing silicon.
  • Watch retirement counters. nvidia-smi -q -d PAGE_RETIREMENT shows pending and retired pages; pending retirements often mean an imminent reboot-driven remap.
  • Collect evidence before rebooting. nvidia-bug-report.sh plus dmesg -T preserves the state that a reboot destroys.
  • Export metrics continuously. DCGM Exporter to Prometheus and Grafana gives you the baseline that makes an anomaly visible at all.

Worked example

A node drops out of a multi-day job. dmesg -T | grep -i xid shows Xid 79 at the failure timestamp, and nvidia-smi no longer lists GPU 3. Reasoning: the device vanished from the PCIe bus rather than returning a compute error, so the fault is below the driver - power delivery, riser, or slot. Reseat and swap the riser first; if the Xid returns on the same slot with a different card, suspect the slot or baseboard, not the GPU.

Traps that cost marks

RMA-ing on a single Xid 13 without checking the application log; running dcgmi diag -r 3 on a production node mid-job; rebooting before capturing nvidia-bug-report.sh output.