NVIDIA NCP-AII cheat sheet

HGX/DGX baseboard GPUs connect to the CPU complex through PCIe switches (NVSwitch handles GPU-to-GPU); confirm every SXM module is fully seated and torqued per the vendor spec before power-on.

GPU-to-GPU NVLink requires matched SXM modules on the same baseboard; mixing GPU SKUs or firmware revisions on one baseboard causes link training failures.

Thermal design: inlet air target is typically 18-27 C; verify airflow direction of every fan module matches the chassis arrow and that blank fillers occupy empty drive and PCIe bays.

Power: size the PDU and circuit for peak TDP of all GPUs plus CPUs; a 8-GPU HGX node can draw 6-10 kW, so confirm redundant feeds and per-outlet ratings before racking.

DIMM population: follow the vendor population table exactly; unbalanced or partially populated channels reduce memory bandwidth and can prevent POST on server platforms.

ECC and memory training: first boot after DIMM changes may take several minutes for memory training; do not cycle power during training.

Cabling discipline: label every NVLink bridge, PCIe aux power, and fabric cable; route power and data separately and verify each connector clicks and the retention latch engages.

Post-installation: run BMC/IPMI sensor sweep (temps, fan RPM, PSU status, leak/flow sensors if liquid-cooled) and confirm no SEL entries before installing software.

Verify enumeration with nvidia-smi -L and lspci | grep -i nvidia; a GPU missing from lspci is a hardware/BIOS issue (Above 4G Decoding, Resizable BAR, PCIe slot bifurcation), not a driver issue.

Persistence mode (nvidia-smi -pm 1) keeps the driver loaded, removing per-process init latency; MIG mode changes require a GPU reset or reboot to take effect.

MIG: enable with nvidia-smi -mig 1, then create instances with nvidia-smi mig -cgi -C; each GI/CI combination has fixed SM and memory slices and cannot be resized without destroying instances.

GPU clocks: nvidia-smi -lgc locks graphics clock, -lmc locks memory clock; use nvidia-smi -q -d CLOCK,PERFORMANCE to inspect throttle reasons (SW Power Cap, HW Slowdown, Thermal).

ECC: enabling ECC on data-center GPUs reserves memory and requires a reboot; check with nvidia-smi -q -d ECC and clear volatile errors with nvidia-smi -c after maintenance.

NVLink health: nvidia-smi nvlink -s shows per-link speed and status; -e enables error counters; degraded links usually indicate seating, baseboard, or firmware mismatch.

Topology: nvidia-smi topo -m reveals NV#/PIX/PXB/SYS/NODE relationships; place communicating ranks on NV# links first, then PIX/PXB, and avoid SYS (cross-socket) for all-reduce-heavy work.

Firmware: GPU and NVSwitch VBIOS plus BMC firmware must match the qualified bundle; mismatched versions cause link-training or thermal-control faults.

InfiniBand fabric: use ibstat/ibstatus for port state (ACTIVE vs INIT/DOWN), iblinkinfo for link width/speed, and perfquery for error counters; rising symbol errors point to a bad cable or transceiver.

Subnet manager: an IB fabric needs exactly one active SM per subnet; check with sminfo or ibdiagnet; without an SM, ports stay INIT and no traffic flows.

RoCEv2 requires lossless Ethernet: configure PFC on priority 3 (or the chosen lossless class) and ECN with DCQCN; verify with counters (pause frames, ECN marked packets) rather than assuming configuration took effect.

MTU: set the same MTU (typically 9000 for jumbo) end-to-end on the fabric; a mismatch causes fragmentation or silent drops for large RDMA messages.

Rail-optimized topology: each GPU rail maps to a separate leaf switch; a single rail failure degrades but does not stop all-reduce, so monitor per-rail counters.

GPUDirect RDMA: requires nvidia-peermem or the in-tree DMA-BUF path plus the correct HCA; verify with ibv_devinfo and a bandwidth test such as perftest (ib_write_bw) between nodes.

Cable/transceiver matching: use the vendor-qualified DAC/AOC/optics for the port speed; mixing 100G and 400G optics on the same link forces the lower speed or fails link training.

Diagnostics order: physical (LEDs, transceiver, cable) -> link (speed/width) -> L2 (FEC, pause, ECN) -> L3 (routing, MTU) -> application (perftest).

Local scratch: NVMe RAID 0 maximizes throughput for dataset staging; RAID 1/5 protects metadata but costs write throughput, so separate fast scratch from durable storage.

GPUDirect Storage enables direct NVMe-to-GPU DMA, bypassing CPU bounce buffers; it requires a supported filesystem (e.g., Lustre, NFS with the right stack) and the nvidia-fs kernel module.

Parallel filesystems (Lustre, GPFS, Weka) are needed when aggregate bandwidth exceeds a single node's NIC; size by GB/s per GPU, not by capacity alone.

Checkpointing: write checkpoints to durable shared storage, not local scratch; align checkpoint interval to expected failure rate (MTBF) and job runtime.

Mount options: use noatime to cut metadata writes, and verify that NFS mount options and rsize/wsize match the fabric MTU for throughput.

Monitoring: watch per-device IOPS, latency, and queue depth (iostat -x, nvme smart-log); rising latency at fixed queue depth signals saturation or a failing device.

NVMe health: nvme smart-log reports media errors, wear level, and temperature; replace devices with growing uncorrectable counts before they drop out of a RAID set.

Capacity planning: reserve headroom (commonly 20%+) on shared filesystems; full parallel filesystems can stall metadata servers and hang jobs.

Install order: OS and kernel headers -> GPU driver -> CUDA toolkit (optional on hosts that only run containers) -> container runtime -> NVIDIA Container Toolkit -> orchestration.

Driver/CUDA compatibility: the driver must be at least the version required by the CUDA runtime; check the CUDA compatibility matrix rather than guessing; the container's CUDA runtime must not exceed the host driver's supported level.

Pre-install checks: disable Nouveau (or the in-tree GPU driver), blacklist it in modprobe.d, and ensure Secure Boot MOK enrollment or disabled Secure Boot, or the module will not load.

Verify driver install with nvidia-smi and by checking /proc/driver/nvidia/version; dmesg should show no NVRM errors.

Container toolkit: configure /etc/docker/daemon.json with the nvidia runtime, restart dockerd, then test with docker run --rm --gpus all nvidia/cuda:12.x-base nvidia-smi.

In containers, use the CUDA compat package or a matching base image; do not install a host driver inside a container.

Version pinning: pin driver, CUDA, cuDNN, NCCL, and framework versions in images; floating tags cause nondeterministic failures across nodes.

Upgrades: drain the node, stop workloads, upgrade driver and firmware as a bundle, reboot, re-run validation (nvidia-smi, NCCL test, health check) before returning it to the pool.

Kubernetes GPU scheduling: install the NVIDIA device plugin; it advertises nvidia.com/gpu resources; pods request GPUs in resources.limits and cannot share a GPU unless MIG or time-slicing is configured.

MIG in Kubernetes: use the MIG manager or the device plugin's MIG strategy (mixed or single); each MIG instance is advertised as a separate schedulable resource.

Slurm: configure Gres=gpu:: in slurm.conf and gres.conf per node; jobs request with --gres=gpu:; use cgroup constraints to enforce GPU isolation.

Topology-aware scheduling: use node labels and affinity to keep tightly coupled ranks on NVLink-connected GPUs and on the same leaf switch where possible.

Gang scheduling: distributed jobs need all-or-nothing allocation; use Slurm's default behavior or a Kubernetes coscheduler/volcano to avoid partial allocations that deadlock.

Health checks: integrate node health checks (DCGM diagnostics, NCCL smoke test) into the scheduler so unhealthy nodes are cordoned or drained automatically.

Resource quotas and namespaces: cap GPU quota per team; without quotas a single job can monopolize the pool.

Job lifecycle: set time limits, preemption policy, and checkpoint-on-preempt so long training jobs survive eviction.

Baseline metrics: GPU utilization, memory used, SM/memory clock, temperature, power draw, ECC errors, NVLink/XID errors, and PCIe replay counters; capture a baseline before and after changes.

DCGM: dcgmi discovery -l lists GPUs; dcgmi diag -r runs health checks; dcgm-exporter exposes metrics to Prometheus and Grafana dashboards.

XID errors: XID 79 (GPU fell off the bus) usually means power, thermal, or seating issues; XID 48 indicates double-bit ECC; look up the XID code before replacing hardware.

NCCL validation: run nccl-tests (all_reduce_perf) across nodes; low bus bandwidth points to topology, MTU, or rail misconfiguration, not to the framework.

Symptom -> layer: job hangs -> check NCCL/IB counters; slow step time -> check GPU clocks/throttle reasons and data loader; OOM -> check memory fragmentation and batch size.

Logs: dmesg for NVRM/XID, journalctl for services, nvidia-smi -q for detailed state, and BMC SEL for hardware events; correlate timestamps across all four.

Throttle reasons from nvidia-smi -q -d PERFORMANCE distinguish thermal, power, and hardware slowdowns; sustained SW Power Cap is expected, HW Slowdown is not.

Validation gate: after any change, re-run nvidia-smi, DCGM diag, and a multi-node NCCL bandwidth test before marking the node healthy.

Least privilege: do not run training jobs as root; use containers with dropped capabilities, read-only root filesystems where possible, and dedicated service accounts for orchestrators.

Firmware and driver supply chain: download only from vendor portals, verify checksums/signatures, and keep a signed bill of materials per node image.

Secure Boot: either enroll the NVIDIA MOK key or disable Secure Boot consistently across the fleet; mixed states cause driver load failures on some nodes only.

Network isolation: separate management, storage, and compute fabrics; restrict BMC/IPMI to a management VLAN and never expose it to the data network.

Secrets: inject credentials via Kubernetes secrets or a vault, never bake them into images or job scripts; rotate keys on a schedule.

Reliability: use redundant PSUs on separate circuits, RAID or replicated storage for checkpoints, and N+1 fabric paths; test failover rather than assuming it works.

Change management: stage driver/firmware upgrades on a canary node, validate, then roll out in waves with a documented rollback path.

Auditability: centralize logs (syslog, BMC SEL, Kubernetes audit) with retention long enough to investigate incidents; alert on XID, ECC, and PSU events.

Enumeration check: nvidia-smi -L should list every expected GPU with the correct product name; missing GPUs indicate BIOS/PCIe configuration or seating problems.

MIG instance creation: nvidia-smi mig -lgip lists profiles, -cgi creates GPU instances, -cci creates compute instances; destroy with -dci/-dgi before reconfiguring.

Clock locking: nvidia-smi -lgc <min,max> and -lmc stabilize performance for benchmarking; reset with -rgc and -rmc.

ECC state changes require a GPU reset; plan them into maintenance windows because they interrupt running jobs.

NVLink diagnostics: nvidia-smi nvlink -s for status, -e to enable error counters, -c to clear; persistent errors after reseating indicate baseboard or module replacement.

Topology mapping: nvidia-smi topo -m; NV# is fastest, then PIX/PXB, then SYS/NODE; rank placement should follow the fastest available link.

Throttle inspection: nvidia-smi -q -d PERFORMANCE lists active throttle reasons; distinguish expected power capping from thermal or hardware slowdowns.

Firmware alignment: keep GPU VBIOS, NVSwitch, and BMC firmware on the vendor-qualified bundle; mismatches cause link and thermal faults.

Port state: ibstat/ibstatus should show ACTIVE with the expected width and speed; INIT or DOWN means SM, cable, or configuration problems.

Fabric discovery: iblinkinfo shows the topology and any links at reduced width/speed; ibdiagnet produces a full fabric health report including SM and error counters.

SM presence: one active subnet manager per IB subnet; verify with sminfo; without it, ports never reach ACTIVE.

RoCEv2 lossless config: PFC on the chosen priority plus ECN marking with DCQCN; verify pause and ECN counters on switches and NICs.

MTU consistency: set 9000 end-to-end for jumbo frames; mismatches fragment or drop large RDMA messages.

GPUDirect RDMA verification: ibv_devinfo lists the HCA; run ib_write_bw between nodes to confirm bandwidth and that traffic bypasses the CPU.

Rail topology: map each GPU rail to a leaf switch; monitor per-rail counters so a single failed rail is visible before it becomes a job failure.

Diagnostic order: physical -> link speed/width -> L2 (FEC, pause, ECN) -> L3 (routing, MTU) -> application (perftest).

Scratch vs durable: local NVMe RAID 0 for staging, shared parallel filesystem for datasets and checkpoints; never checkpoint to local scratch.

GPUDirect Storage: requires the nvidia-fs module and a supported filesystem; verify with gdscheck before relying on direct DMA paths.

Filesystem choice: Lustre/GPFS/Weka for multi-node bandwidth beyond one NIC; NFS only for small metadata or low-throughput shares.

Mount options: noatime reduces metadata writes; verify rsize/wsize and fabric MTU alignment for NFS throughput.

Health: nvme smart-log for media errors, wear, and temperature; replace devices with growing uncorrectable counts proactively.

Monitoring: iostat -x for per-device utilization, latency, and queue depth; rising latency at fixed queue depth signals saturation.

Capacity: keep 20%+ headroom on shared filesystems; full parallel filesystems can stall metadata servers and hang jobs.

Checkpoint interval: balance against MTBF and job runtime; store on redundant, replicated storage with tested restore.

Install order: OS and kernel headers -> driver -> optional CUDA toolkit -> container runtime -> NVIDIA Container Toolkit -> orchestration.

Compatibility: the host driver must support the CUDA runtime used in containers; check the CUDA compatibility matrix and never install a driver inside a container.

Pre-install: blacklist Nouveau, handle Secure Boot MOK enrollment, and confirm kernel headers match the running kernel before building the module.

Verification: nvidia-smi, /proc/driver/nvidia/version, and dmesg free of NVRM errors.

Container runtime: add the nvidia runtime to /etc/docker/daemon.json, restart dockerd, test with docker run --rm --gpus all nvidia/cuda:12.x-base nvidia-smi.

Version pinning: pin driver, CUDA, cuDNN, NCCL, and framework versions in images; floating tags cause nondeterministic cross-node failures.

Upgrade procedure: drain node, stop workloads, upgrade driver and firmware as a bundle, reboot, re-run validation before returning to the pool.

CUDA compat package: allows newer CUDA runtimes on older drivers within the supported forward-compatibility window; still verify the matrix.

Kubernetes: the NVIDIA device plugin advertises nvidia.com/gpu; pods request GPUs in resources.limits; sharing requires MIG or time-slicing configuration.

MIG scheduling: the device plugin MIG strategy (mixed or single) determines how MIG instances appear as schedulable resources.

Slurm: Gres=gpu:: in slurm.conf plus gres.conf per node; jobs request with --gres=gpu:; use cgroups for isolation.

Topology-aware placement: node labels and affinity keep tightly coupled ranks on NVLink-connected GPUs and on the same leaf switch.

Gang scheduling: distributed jobs need all-or-nothing allocation; use Slurm defaults or a Kubernetes coscheduler/volcano to avoid partial-allocation deadlocks.

Health integration: feed DCGM diagnostics and NCCL smoke tests into the scheduler so unhealthy nodes are cordoned or drained automatically.

Quotas: cap GPU quota per namespace or account; without quotas one job can monopolize the pool.

Lifecycle: set time limits, preemption policy, and checkpoint-on-preempt so long training jobs survive eviction.

Baseline metrics: GPU utilization, memory, clocks, temperature, power, ECC, NVLink/XID errors, and PCIe replay counters; capture before and after changes.

DCGM: dcgmi discovery -l, dcgmi diag -r , and dcgm-exporter to Prometheus/Grafana.

XID codes: 79 (GPU fell off bus) points to power, thermal, or seating; 48 is double-bit ECC; look up the code before replacing hardware.

NCCL validation: nccl-tests all_reduce_perf across nodes; low bus bandwidth indicates topology, MTU, or rail misconfiguration.

Symptom mapping: hangs -> NCCL/IB counters; slow steps -> clocks/throttle and data loader; OOM -> fragmentation and batch size.

Log correlation: dmesg (NVRM/XID), journalctl (services), nvidia-smi -q (state), BMC SEL (hardware); align timestamps across all four.

Throttle reasons: SW Power Cap is expected; HW Slowdown indicates thermal or hardware problems requiring action.

Validation gate: after any change, re-run nvidia-smi, DCGM diag, and a multi-node NCCL bandwidth test before marking the node healthy.

Least privilege: no root training jobs; containers with dropped capabilities, read-only root filesystems where possible, dedicated service accounts.

Supply chain: download firmware/drivers only from vendor portals, verify checksums/signatures, keep a signed bill of materials per node image.

Secure Boot: enroll the NVIDIA MOK key or disable Secure Boot consistently fleet-wide; mixed states cause driver load failures on some nodes only.

Network isolation: separate management, storage, and compute fabrics; restrict BMC/IPMI to a management VLAN.

Secrets: inject via Kubernetes secrets or a vault; never bake into images or job scripts; rotate on a schedule.

Reliability: redundant PSUs on separate circuits, RAID or replicated checkpoint storage, N+1 fabric paths; test failover rather than assuming it works.

Change management: canary node first, validate, then wave rollout with a documented rollback path.

Auditability: centralize syslog, BMC SEL, and Kubernetes audit logs with sufficient retention; alert on XID, ECC, and PSU events.

SXM baseboard: every GPU module must be fully seated and torqued to spec; NVSwitch provides GPU-to-GPU links while PCIe switches connect GPUs to CPUs.

Matching: use identical GPU SKUs and firmware on one baseboard; mixing causes NVLink training failures.

Thermal: inlet air 18-27 C typical; verify fan airflow direction and install blank fillers in empty bays.

Power: size PDU and circuits for peak GPU+CPU TDP; an 8-GPU HGX node can draw 6-10 kW; confirm redundant feeds.

DIMMs: follow the population table exactly; unbalanced channels reduce bandwidth and can prevent POST.

Memory training: first boot after DIMM changes can take minutes; do not cycle power during training.

Cabling: label NVLink bridges, aux power, and fabric cables; route power and data separately and confirm retention latches engage.

Post-install: BMC sensor sweep (temps, fan RPM, PSU, leak/flow sensors) with no SEL entries before software installation.

Enumeration: nvidia-smi -L and lspci | grep -i nvidia; a GPU absent from lspci is a BIOS/PCIe issue (Above 4G Decoding, Resizable BAR, bifurcation).

Persistence mode (nvidia-smi -pm 1) removes per-process init latency; MIG mode changes need a GPU reset or reboot.

MIG: nvidia-smi -mig 1 to enable; nvidia-smi mig -cgi -C to create; slices are fixed and cannot be resized without destroying instances.

Clocks: -lgc locks graphics clock, -lmc locks memory clock; inspect throttle reasons with nvidia-smi -q -d CLOCK,PERFORMANCE.

ECC: enabling reserves memory and requires reboot; check with nvidia-smi -q -d ECC; plan changes into maintenance windows.

NVLink: nvidia-smi nvlink -s for status, -e for error counters, -c to clear; persistent errors after reseating indicate hardware replacement.

Topology: nvidia-smi topo -m; NV# fastest, then PIX/PXB, then SYS/NODE; place communicating ranks on the fastest link.

Firmware: keep GPU VBIOS, NVSwitch, and BMC on the vendor-qualified bundle; mismatches cause link and thermal faults.

Port state: ibstat/ibstatus should show ACTIVE at expected width and speed; INIT/DOWN indicates SM, cable, or configuration problems.

Discovery: iblinkinfo for topology and reduced-width links; ibdiagnet for a full fabric health report.

SM: exactly one active subnet manager per IB subnet; verify with sminfo; without it ports stay INIT.

RoCEv2: PFC on the chosen priority plus ECN with DCQCN; verify pause and ECN-marked counters rather than assuming config took effect.

MTU: 9000 end-to-end for jumbo frames; mismatches fragment or drop large RDMA messages.

GPUDirect RDMA: requires nvidia-peermem or the DMA-BUF path plus the correct HCA; verify with ibv_devinfo and ib_write_bw.

Rail topology: each GPU rail maps to a leaf switch; monitor per-rail counters to catch a single failed rail early.

Diagnostic order: physical -> link speed/width -> L2 (FEC, pause, ECN) -> L3 (routing, MTU) -> application (perftest).

Local scratch: NVMe RAID 0 for maximum staging throughput; RAID 1/5 protects metadata but reduces write throughput.

GPUDirect Storage: direct NVMe-to-GPU DMA bypassing CPU bounce buffers; needs nvidia-fs and a supported filesystem.

Parallel filesystems (Lustre, GPFS, Weka) when aggregate bandwidth exceeds a single NIC; size by GB/s per GPU.

Checkpointing: write to durable shared storage, not local scratch; align interval to MTBF and job runtime.

Mount options: noatime cuts metadata writes; match rsize/wsize and fabric MTU for NFS throughput.

Monitoring: iostat -x for IOPS, latency, queue depth; rising latency at fixed queue depth signals saturation or a failing device.

NVMe health: nvme smart-log for media errors, wear, temperature; replace devices with growing uncorrectable counts.

Capacity: keep 20%+ headroom on shared filesystems; full parallel filesystems can stall metadata servers and hang jobs.

Order: OS and kernel headers -> driver -> optional CUDA toolkit -> container runtime -> NVIDIA Container Toolkit -> orchestration.

Compatibility: host driver must support the container's CUDA runtime; check the CUDA compatibility matrix; never install a driver inside a container.

Pre-install: blacklist Nouveau, handle Secure Boot MOK enrollment, confirm kernel headers match the running kernel.

Verify: nvidia-smi, /proc/driver/nvidia/version, and dmesg without NVRM errors.

Container test: docker run --rm --gpus all nvidia/cuda:12.x-base nvidia-smi after configuring the nvidia runtime in daemon.json.

Pinning: fix driver, CUDA, cuDNN, NCCL, and framework versions in images; floating tags cause nondeterministic failures.

Upgrade: drain node, stop workloads, upgrade driver and firmware as a bundle, reboot, re-validate before returning to the pool.

CUDA compat package: runs newer CUDA runtimes on older drivers within the forward-compatibility window; verify the matrix.

Kubernetes: NVIDIA device plugin advertises nvidia.com/gpu; request GPUs in resources.limits; sharing needs MIG or time-slicing.

MIG strategy: mixed or single determines how MIG instances appear as schedulable resources.

Slurm: Gres=gpu:: in slurm.conf and gres.conf; jobs use --gres=gpu:; cgroups enforce isolation.

Topology-aware: node labels and affinity keep coupled ranks on NVLink-connected GPUs and the same leaf switch.

Gang scheduling: all-or-nothing allocation via Slurm defaults or a Kubernetes coscheduler/volcano prevents partial-allocation deadlocks.

Health integration: DCGM diagnostics and NCCL smoke tests feed the scheduler so unhealthy nodes are cordoned or drained.

Quotas: cap GPU quota per namespace or account so one job cannot monopolize the pool.

Lifecycle: time limits, preemption policy, and checkpoint-on-preempt protect long training jobs.

Baseline: utilization, memory, clocks, temperature, power, ECC, NVLink/XID errors, PCIe replay counters; capture before and after changes.

DCGM: dcgmi discovery -l, dcgmi diag -r , dcgm-exporter to Prometheus/Grafana.

XID: 79 (fell off bus) suggests power, thermal, or seating; 48 is double-bit ECC; identify the code before replacing hardware.

NCCL: nccl-tests all_reduce_perf across nodes; low bus bandwidth indicates topology, MTU, or rail misconfiguration.

Symptom mapping: hangs -> NCCL/IB counters; slow steps -> clocks/throttle and data loader; OOM -> fragmentation and batch size.

Logs: dmesg (NVRM/XID), journalctl (services), nvidia-smi -q (state), BMC SEL (hardware); correlate timestamps.

Throttle reasons: SW Power Cap expected; HW Slowdown indicates thermal or hardware problems.

Validation gate: re-run nvidia-smi, DCGM diag, and multi-node NCCL bandwidth test after any change.

Least privilege: no root training jobs; drop capabilities, read-only root filesystems where possible, dedicated service accounts.

Supply chain: vendor-only firmware/driver downloads, checksum/signature verification, signed bill of materials per node image.

Secure Boot: enroll NVIDIA MOK or disable consistently fleet-wide; mixed states cause driver load failures on some nodes.

Isolation: separate management, storage, and compute fabrics; restrict BMC/IPMI to a management VLAN.

Secrets: Kubernetes secrets or a vault; never in images or job scripts; rotate on a schedule.

Reliability: redundant PSUs on separate circuits, RAID or replicated checkpoint storage, N+1 fabric paths; test failover.

Change management: canary node, validate, then wave rollout with a documented rollback path.

Audit: centralize syslog, BMC SEL, and Kubernetes audit logs; alert on XID, ECC, and PSU events.

SXM baseboard: fully seat and torque each GPU module; NVSwitch handles GPU-to-GPU, PCIe switches connect GPUs to CPUs.

Matching: identical GPU SKUs and firmware on one baseboard; mixing causes NVLink training failures.

Thermal: inlet air 18-27 C; verify fan airflow direction and fill empty drive/PCIe bays with blanks.

Power: size PDU and circuits for peak GPU+CPU TDP; 8-GPU HGX nodes can draw 6-10 kW; confirm redundant feeds.

DIMMs: follow the population table; unbalanced channels cut bandwidth and can prevent POST.

Memory training: first boot after DIMM changes takes minutes; do not cycle power during training.

Cabling: label NVLink bridges, aux power, and fabric cables; confirm retention latches engage.

Post-install: BMC sensor sweep (temps, fan RPM, PSU, leak sensors) with no SEL entries before software.

Enumeration: nvidia-smi -L and lspci | grep -i nvidia; absent from lspci means BIOS/PCIe configuration or seating.

Persistence mode (nvidia-smi -pm 1) removes init latency; MIG mode changes need a GPU reset or reboot.

MIG: nvidia-smi -mig 1 enables; nvidia-smi mig -cgi -C creates; slices are fixed until destroyed.

Clocks: -lgc for graphics clock, -lmc for memory clock; reset with -rgc and -rmc.

ECC: enabling reserves memory and requires reboot; check nvidia-smi -q -d ECC; schedule in maintenance windows.

NVLink: nvidia-smi nvlink -s status, -e error counters, -c clear; persistent errors after reseating mean replacement.

Topology: nvidia-smi topo -m; NV# > PIX/PXB > SYS/NODE; place communicating ranks accordingly.

Firmware: GPU VBIOS, NVSwitch, and BMC on the vendor-qualified bundle; mismatches cause link and thermal faults.

ibstat/ibstatus: ACTIVE with expected width and speed; INIT/DOWN means SM, cable, or configuration issues.

iblinkinfo shows topology and reduced-width links; ibdiagnet gives a full fabric health report.

SM: exactly one active subnet manager per IB subnet; check sminfo; without it ports stay INIT.

RoCEv2: PFC on the chosen priority plus ECN/DCQCN; verify pause and ECN counters on switches and NICs.

MTU: 9000 end-to-end; mismatches fragment or drop large RDMA messages.

GPUDirect RDMA: nvidia-peermem or DMA-BUF plus the correct HCA; verify with ibv_devinfo and ib_write_bw.

Rails: each GPU rail maps to a leaf switch; per-rail counters reveal a single failed rail early.

Order: physical -> link speed/width -> L2 (FEC, pause, ECN) -> L3 (routing, MTU) -> application (perftest).

Scratch: NVMe RAID 0 for staging throughput; RAID 1/5 protects metadata but lowers write throughput.

GPUDirect Storage: NVMe-to-GPU DMA bypassing CPU bounce buffers; requires nvidia-fs and a supported filesystem.

Parallel filesystems (Lustre, GPFS, Weka) when bandwidth exceeds one NIC; size by GB/s per GPU.

Checkpoints: durable shared storage, never local scratch; interval balanced against MTBF.

Mounts: noatime reduces metadata writes; align rsize/wsize and MTU for NFS throughput.

Monitoring: iostat -x for IOPS, latency, queue depth; latency rising at fixed queue depth signals saturation.

NVMe: nvme smart-log for media errors, wear, temperature; replace devices with growing uncorrectable counts.

Capacity: keep 20%+ headroom; full parallel filesystems can stall metadata servers and hang jobs.

Order: OS and kernel headers -> driver -> optional CUDA toolkit -> container runtime -> NVIDIA Container Toolkit -> orchestration.

Compatibility: host driver must support the container CUDA runtime; check the CUDA matrix; no driver inside containers.

Pre-install: blacklist Nouveau, Secure Boot MOK enrollment, kernel headers matching the running kernel.

Verify: nvidia-smi, /proc/driver/nvidia/version, dmesg without NVRM errors.

Container test: docker run --rm --gpus all nvidia/cuda:12.x-base nvidia-smi after setting the nvidia runtime in daemon.json.

Pinning: fix driver, CUDA, cuDNN, NCCL, and framework versions in images; floating tags cause cross-node failures.

Upgrade: drain, stop workloads, upgrade driver and firmware as a bundle, reboot, re-validate.

CUDA compat package: newer CUDA runtime on older drivers within the forward-compatibility window; verify the matrix.

Kubernetes: device plugin advertises nvidia.com/gpu; request in resources.limits; sharing requires MIG or time-slicing.

MIG strategy: mixed or single controls how MIG instances are advertised as resources.

Slurm: Gres=gpu:: in slurm.conf and gres.conf; jobs use --gres=gpu:; cgroups isolate.

Topology-aware: labels and affinity keep coupled ranks on NVLink and the same leaf switch.

Gang scheduling: all-or-nothing allocation via Slurm or a coscheduler/volcano prevents deadlocks.

Health: DCGM diagnostics and NCCL smoke tests cordon or drain unhealthy nodes automatically.

Quotas: cap GPU quota per namespace or account to prevent monopolization.

Lifecycle: time limits, preemption, checkpoint-on-preempt protect long training jobs.

Baseline: utilization, memory, clocks, temperature, power, ECC, NVLink/XID errors, PCIe replay; capture before and after changes.

DCGM: dcgmi discovery -l, dcgmi diag -r , dcgm-exporter to Prometheus/Grafana.

XID: 79 (fell off bus) suggests power, thermal, or seating; 48 is double-bit ECC; identify before replacing hardware.

NCCL: nccl-tests all_reduce_perf; low bus bandwidth indicates topology, MTU, or rail misconfiguration.

Symptom mapping: hangs -> NCCL/IB counters; slow steps -> clocks/throttle and data loader; OOM -> fragmentation and batch size.

Logs: dmesg (NVRM/XID), journalctl, nvidia-smi -q, BMC SEL; correlate timestamps.

Throttle reasons: SW Power Cap expected; HW Slowdown signals thermal or hardware problems.

Validation gate: re-run nvidia-smi, DCGM diag, and multi-node NCCL bandwidth test after any change.

Least privilege: no root training jobs; drop capabilities, read-only root filesystems where possible, dedicated service accounts.

Supply chain: vendor-only downloads, checksum/signature verification, signed bill of materials per node image.

Secure Boot: enroll NVIDIA MOK or disable consistently; mixed states cause driver load failures on some nodes.

Isolation: separate management, storage, and compute fabrics; BMC/IPMI on a management VLAN only.

Secrets: Kubernetes secrets or a vault; never in images or scripts; rotate on a schedule.

Reliability: redundant PSUs on separate circuits, RAID or replicated checkpoints, N+1 fabric paths; test failover.

Change management: canary node, validate, wave rollout with documented rollback.

Audit: centralize syslog, BMC SEL, and Kubernetes audit logs; alert on XID, ECC, and PSU events.

SXM baseboard: fully seat and torque GPU modules; NVSwitch handles GPU-to-GPU, PCIe switches connect GPUs to CPUs.

Matching: identical GPU SKUs and firmware per baseboard; mixing causes NVLink training failures.

Thermal: inlet air 18-27 C; verify fan airflow direction and fill empty bays with blanks.

Power: size PDU and circuits for peak GPU+CPU TDP; 8-GPU HGX nodes can draw 6-10 kW; confirm redundant feeds.

DIMMs: follow the population table; unbalanced channels reduce bandwidth and can prevent POST.

Memory training: first boot after DIMM changes takes minutes; do not cycle power during training.

Cabling: label NVLink bridges, aux power, and fabric cables; confirm retention latches engage.

Post-install: BMC sensor sweep (temps, fan RPM, PSU, leak sensors) with no SEL entries before software.