I85PREP
⌘ K
NVIDIA NCP-AII study materials
NVIDIA NCP-AII study resources

NVIDIA NCP-AII: Security, Reliability, and Operational Best Practices

NVIDIA NCP-AII · Work through your exam, one topic at a time.

All guide sections

Security, Reliability, and Operational Best Practices

What this domain tests

Running an AI cluster that stays up and stays contained: secure boot and module signing, multi-tenant isolation, firmware baselines, out-of-band management, redundancy, and change management for driver and fabric upgrades.

Core rules

  • Secure boot requires signed modules. With UEFI Secure Boot enabled, the NVIDIA kernel modules must be signed and the key enrolled via MOK, otherwise the driver silently fails to load after a kernel update. Signing must be repeated for each new kernel or handled through DKMS automation.
  • Isolation choices. MIG gives memory and fault isolation with dedicated SMs. Confidential Computing on Hopper protects data in use, but it restricts or disables MIG and carries a performance cost - plan capacity accordingly.
  • Out-of-band separation. BMC/IPMI must sit on a management network separate from the in-band fabric, with credentials rotated and firmware patched; a compromised BMC owns the node.
  • Firmware and driver baselines. Define a validated matrix of GPU VBIOS, NIC firmware, driver, and Fabric Manager versions, then roll changes canary-first: drain a node, upgrade, run dcgmi diag -r 2, verify a NCCL all-reduce, and only then proceed.
  • RAS posture. Keep ECC on, monitor row remapping, and pair long jobs with frequent async checkpoints so a node loss costs minutes, not days.

Worked example

After upgrading a DGX-class node from driver 550 to 560, four GPUs disappear and dmesg floods with NVSwitch errors. Reasoning: NVSwitch is managed by Fabric Manager, whose version must match the driver exactly; the driver moved and Fabric Manager did not. The fix is to install the matching nvidia-fabricmanager build and restart the service - not to reseat hardware or roll back blindly. The general rule: driver and Fabric Manager upgrade as one unit.

Traps that cost marks

Upgrading the driver alone on NVSwitch systems; enabling Confidential Computing and then expecting MIG partitioning to work unchanged; treating BMC access as an afterthought.