I85PREP
⌘ K
NVIDIA NCP-AII study materials
NVIDIA NCP-AII study resources

NVIDIA NCP-AII: Software Stack Installation and Management (drivers, CUDA, containers)

NVIDIA NCP-AII · Work through your exam, one topic at a time.

All guide sections

Software Stack Installation and Management (drivers, CUDA, containers)

What this domain tests

The install order and version contracts that make a GPU node usable: kernel driver, CUDA toolkit, container runtime integration, Fabric Manager, and DCGM, plus the compatibility rules that tie them together.

Core rules

  • Install order. OS and kernel headers, firmware/BIOS, GPU driver (DKMS or precompiled), CUDA toolkit, container toolkit, then Fabric Manager and DCGM. Skipping kernel headers produces a driver that builds but does not load.
  • Version contracts. nvidia-smi reports the maximum CUDA runtime the installed driver supports - that is a ceiling, not an installed toolkit version. Minor-version compatibility lets a 12.x app run on a 12.y driver within limits; crossing major versions requires the CUDA forward compatibility package or a driver upgrade.
  • Fabric Manager must match the driver exactly, including the minor build. This is the single most common post-upgrade breakage on HGX and DGX systems.
  • Containers. The NVIDIA Container Toolkit injects devices and driver libraries; configure it with nvidia-ctk runtime configure --runtime=docker, then verify with docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi.
  • Slurm sites typically use Enroot plus Pyxis, where --container-image needs --container-mounts for datasets and --nv-equivalent GPU passthrough.

Worked example

An application built against CUDA 12.4 fails at startup with "CUDA driver version is insufficient for CUDA runtime version". nvidia-smi reports CUDA Version 12.2 in the header. The reasoning: the driver's 12.2 ceiling cannot satisfy a 12.4 runtime, and this is a major-version gap, not a minor one. Two valid fixes - upgrade the driver to one that supports 12.4, or install the CUDA forward compatibility package inside the container. Restarting the app changes nothing.

Traps that cost marks

Reading the CUDA version from nvidia-smi as if it were the installed toolkit; upgrading the driver but not Fabric Manager; mixing a runfile install over a package-manager install and ending up with two driver versions.