NVIDIA NCP-AII flashcards

32 cards. Read the term and the definition here. The deck at the top of this page is optional.

SXM baseboard seating

Each SXM GPU module must be fully seated and torqued to the vendor specification; partial seating causes NVLink training failures and intermittent link-down events visible in nvidia-smi nvlink -s.

Above 4G Decoding

A BIOS setting that lets the system map large PCIe BARs above 4 GB; if disabled, GPUs may fail to enumerate or show reduced BAR sizes in lspci, breaking driver initialization.

Memory training delay

After DIMM changes, server platforms retrain memory on first boot, which can take several minutes; power cycling during training can corrupt the configuration and prevent POST.

Persistence mode

Enabled with nvidia-smi -pm 1, it keeps the NVIDIA driver loaded after the last process exits, removing per-process initialization latency on shared GPU nodes.

MIG instance creation

Enable MIG with nvidia-smi -mig 1, then create GPU and compute instances with nvidia-smi mig -cgi and -cci; slices are fixed and must be destroyed before reconfiguration.

GPU throttle reasons

nvidia-smi -q -d PERFORMANCE lists active throttle causes; SW Power Cap is expected under load, while HW Slowdown indicates thermal or hardware problems requiring action.

NVLink error counters

nvidia-smi nvlink -e enables per-link error counters and -s shows link status; persistent errors after reseating modules usually indicate a baseboard or module replacement.

InfiniBand port states

ibstat and ibstatus report port state; ACTIVE means the link is usable, while INIT or DOWN typically indicates a missing subnet manager, bad cable, or transceiver mismatch.

Subnet manager requirement

An InfiniBand subnet needs exactly one active subnet manager; without it, ports never transition to ACTIVE and no fabric traffic flows. Verify with sminfo.

RoCEv2 lossless configuration

RoCEv2 requires PFC on the chosen lossless priority plus ECN marking with DCQCN; verify pause-frame and ECN-marked counters on switches and NICs rather than assuming configuration applied.

GPUDirect RDMA verification

Requires nvidia-peermem or the DMA-BUF path plus a supported HCA; confirm with ibv_devinfo and measure node-to-node bandwidth with ib_write_bw from perftest.

Rail-optimized topology

Each GPU rail connects to its own leaf switch; a single rail failure degrades but does not stop all-reduce, so per-rail counters reveal failures before jobs break.

GPUDirect Storage

Enables direct DMA between NVMe storage and GPU memory, bypassing CPU bounce buffers; it requires the nvidia-fs kernel module and a supported filesystem such as Lustre.

Checkpoint storage placement

Checkpoints must be written to durable shared storage, not local scratch; local NVMe RAID 0 offers throughput but no redundancy, so a node failure loses the checkpoint.

NVMe SMART health

nvme smart-log reports media errors, wear level, and temperature; replace devices with growing uncorrectable error counts before they drop out of a RAID set.

noatime mount option

Mounting with noatime stops the filesystem from recording access times on every read, cutting metadata writes and improving throughput on busy dataset and scratch filesystems.

Nouveau blacklist

The open-source Nouveau driver conflicts with the NVIDIA proprietary driver; blacklist it in modprobe.d and rebuild the initramfs before installing the data-center driver.

NVIDIA Container Toolkit test

After adding the nvidia runtime to /etc/docker/daemon.json and restarting dockerd, run docker run --rm --gpus all nvidia/cuda:12.x-base nvidia-smi to confirm GPU access inside containers.

Driver CUDA compatibility

The host driver must support the CUDA runtime used by containers; check the CUDA compatibility matrix and never install a GPU driver inside a container image.

Version pinning

Pin driver, CUDA, cuDNN, NCCL, and framework versions in container images; floating tags cause nondeterministic failures that appear on some nodes but not others.

Kubernetes device plugin

The NVIDIA device plugin advertises nvidia.com/gpu resources; pods request GPUs in resources.limits, and sharing requires MIG or time-slicing configuration.

Slurm GRES GPU config

Declare Gres=gpu:<type>:<count> in slurm.conf and per-node entries in gres.conf; jobs request GPUs with --gres=gpu:<count>, and cgroups enforce isolation.

Gang scheduling

Distributed training jobs need all-or-nothing allocation; partial allocation deadlocks collectives, so use Slurm defaults or a Kubernetes coscheduler such as volcano.

MIG in Kubernetes

The device plugin MIG strategy, mixed or single, determines how MIG instances are advertised; each instance becomes a separately schedulable GPU resource.

DCGM diagnostics

dcgmi discovery -l lists GPUs and dcgmi diag -r <level> runs health checks; dcgm-exporter publishes metrics to Prometheus for dashboards and alerting.

XID 79

An XID 79 event means the GPU fell off the PCIe bus; it usually indicates power delivery, thermal, or seating problems rather than a software fault.

NCCL bandwidth test

Run nccl-tests all_reduce_perf across nodes; low bus bandwidth points to topology, MTU, or rail misconfiguration rather than a framework problem.

Symptom to layer mapping

Job hangs point to NCCL or IB counters; slow steps point to clocks, throttle reasons, or the data loader; out-of-memory points to fragmentation and batch size.

Secure Boot MOK enrollment

With Secure Boot enabled, the NVIDIA kernel module loads only if the vendor MOK key is enrolled or Secure Boot is disabled; mixed states cause failures on some nodes only.

BMC network isolation

Restrict BMC and IPMI interfaces to a dedicated management VLAN; exposing them on data or compute networks creates a high-privilege attack surface.

Canary upgrade rollout

Stage driver and firmware upgrades on one canary node, validate with nvidia-smi, DCGM diag, and NCCL tests, then roll out in waves with a documented rollback path.

Redundant PSU circuits

Connect redundant power supplies to separate circuits or PDUs so a single feed failure does not take down a multi-kilowatt GPU node mid-training.