NVIDIA NCP-AIO cheat sheet

Rack power density for AI: an 8-GPU HGX/DGX-class node typically draws 6-10 kW; a full rack of such nodes can exceed 100 kW, so liquid cooling (DLC) is required above roughly 30-40 kW per rack where air cooling becomes impractical.

Power usage effectiveness (PUE) = total facility power / IT equipment power. AI halls with DLC can achieve lower PUE than air-cooled halls because coolant loops reject heat more efficiently than CRAC/CRAH units.

Redundancy notation: N = no redundancy; N+1 = one spare component; 2N = fully duplicated; 2(N+1) = duplicated with a spare in each path. AI training clusters are commonly specified at N+1 for power distribution and 2N for upstream utility feeds.

Thermal design: GPU inlet air target is typically 18-27 C per ASHRAE A1/A2 envelopes; exceeding the upper bound triggers GPU clock throttling and reduces sustained training throughput.

Coolant loop facts for DLC: facility water (Tier 3) is isolated from the technology cooling system (TCS) by a CDU; typical supply temperatures are 30-45 C, and leak detection plus flow/pressure monitoring is mandatory.

Weight and floor loading: a fully populated AI rack can weigh over 1,500 kg; raised-floor data centers often need structural reinforcement or slab-mounted racks because point loads exceed standard 8-12 kN/m2 ratings.

Stranded capacity arises when power, cooling, or space is exhausted before the others; AI deployments frequently strand cooling first because air-cooled halls cannot reject the heat density.

Maintenance windows: GPU nodes with ECC and row-remap capability can tolerate correctable memory errors, but uncorrectable errors require node drain; plan rolling drains per rack to preserve cluster quorum.

Reference architecture layers: compute (GPU nodes) -> storage (parallel filesystem) -> management network (BMC/1GbE) -> in-band fabric (InfiniBand or RoCE) -> scheduler/ orchestrator. Each layer is validated before the next is brought up.

Cluster bring-up order: power and cooling commissioning, BMC/out-of-band addressing, firmware baseline, OS image, GPU driver and CUDA, fabric manager, container runtime, then scheduler. Skipping firmware baseline causes nondeterministic failures later.

GPU node acceptance test: run a burn-in (e.g., sustained matrix-multiply or memory bandwidth loop) plus NCCL all-reduce across all GPUs to detect thermal, power, or fabric defects before accepting the node into production.

Firmware consistency matters: mismatched GPU, NIC, and switch firmware across a cluster is a leading cause of intermittent link flaps and training job failures; track a golden firmware bill of materials.

Capacity planning rule: usable GPU-hours = nodes x GPUs per node x hours x utilization factor; plan for 10-20% headroom for failures, maintenance, and fragmentation.

Job scheduler (Slurm/Kubernetes) placement: gang scheduling is required for multi-node training so all ranks start together; partial allocation wastes GPU-hours while waiting for stragglers.

Health checks: monitor GPU temperature, power draw, XID errors, ECC counters, NVLink status, and fabric link errors. An XID error in dmesg indicates a driver or hardware fault on that GPU.

Lifecycle management: use image-based provisioning and configuration management (e.g., Ansible, MAAS) so a failed node can be reimaged to the golden baseline without manual steps.

GPU memory hierarchy: registers -> shared memory / L1 -> L2 -> HBM (global). HBM bandwidth on data-center GPUs is on the order of terabytes per second, far exceeding PCIe host bandwidth.

Occupancy = active warps per SM / maximum warps per SM. Low occupancy from excessive registers or shared memory limits latency hiding; check with the occupancy calculator or profiler.

Roofline model: attainable performance = min(peak FLOPs, arithmetic intensity x peak memory bandwidth). Arithmetic intensity = FLOPs / bytes moved; below the ridge point the kernel is memory-bound.

Tensor cores accelerate mixed-precision matrix multiply (FP16/BF16/FP8 inputs with FP32 accumulate); using them requires dimensions aligned to their tile shapes and appropriate data types.

NVLink provides GPU-to-GPU bandwidth far above PCIe; on multi-GPU nodes, peer-to-peer transfers over NVLink avoid host-memory bounce and are essential for model-parallel training.

MIG (Multi-Instance GPU) partitions a single GPU into isolated slices with dedicated memory and SMs; useful for inference density but not for workloads needing full-GPU NVLink bandwidth.

ECC on HBM protects against bit flips; correctable errors are logged and row-remapped, uncorrectable errors typically require resetting or replacing the GPU. Monitor volatile and aggregate ECC counts.

Profiling workflow: use Nsight Systems for timeline (kernel gaps, sync stalls), Nsight Compute for per-kernel metrics (achieved occupancy, memory throughput, warp stall reasons).

InfiniBand vs RoCE: InfiniBand uses lossless link-level flow control and credit-based congestion control; RoCEv2 runs over Ethernet and requires PFC (priority flow control) and ECN for lossless behavior.

Fat-tree topology: a k-ary fat-tree with k pods supports k^3/4 hosts; oversubscription ratio = downstream bandwidth / upstream bandwidth. AI training fabrics are typically 1:1 (non-blocking) for all-reduce efficiency.

Rail-optimized design: connect GPU index N on every node to the same leaf/rail so all-reduce traffic between matching GPU ranks stays within one hop, reducing hop count and congestion.

NCCL selects transports (NVLink, InfiniBand, RoCE, PCIe, shared memory) automatically; setting NCCL_IB_HCA, NCCL_SOCKET_IFNAME, and NCCL_DEBUG=INFO diagnoses ring/tree construction and transport selection.

All-reduce ring algorithm moves approximately 2(N-1)/N times the message size per rank; bandwidth scales with the slowest link, so a single degraded link throttles the whole collective.

GPUDirect RDMA lets NICs DMA directly to GPU memory, bypassing host memory copies; it requires IOMMU passthrough, peer-to-peer support, and matching NIC/GPU placement on the same PCIe/NVLink domain.

Congestion control: DCQCN (RoCE) and adaptive routing (InfiniBand) mitigate incast; buffer sizing and ECN marking thresholds must be tuned or tail latency spikes during all-reduce.

Link health: monitor symbol errors, CRC errors, and link down events per port; a flapping link causes NCCL to fall back or hang, so set timeouts (NCCL_IB_TIMEOUT) appropriate to fabric size.

Stack layers bottom-up: GPU driver -> CUDA runtime/toolkit -> cuDNN/cuBLAS/NCCL libraries -> framework (PyTorch/TensorFlow) -> container image -> orchestrator. Version compatibility must be validated at each boundary.

CUDA compatibility: the driver must be at least the version required by the CUDA toolkit; minor-version compatibility allows newer toolkits on older drivers within limits, but major versions require a driver upgrade.

Containers: NVIDIA Container Toolkit injects driver libraries and devices into containers; the host driver version must be compatible with the CUDA version baked into the image.

Distributed training parallelism: data parallel replicates the model and splits batches; tensor parallel splits layers within a node (NVLink); pipeline parallel splits layers across nodes. Large models combine all three.

Mixed precision: BF16 has the same exponent range as FP32 with reduced mantissa, avoiding loss-scaling issues of FP16; FP8 on newer GPUs requires per-tensor or per-block scaling.

Checkpointing: save optimizer state, model weights, and RNG state; for large models use sharded/distributed checkpointing to avoid a single writer bottleneck and to allow resharding on restart.

Observability: collect GPU utilization, memory used, SM occupancy, and job-level throughput (samples/sec); low GPU utilization with high data-loader time indicates input pipeline starvation.

Failure handling: set elastic training or checkpoint-restart policies; an unplanned node loss in synchronous data parallel stalls all ranks until the job is restarted from the last checkpoint.

Redundancy and concurrent maintainability determine whether maintenance can proceed without dropping AI workloads; Tier III/IV facilities support concurrent maintainability, Tier I/II do not.

Cooling capacity check: heat load (kW) = IT power draw (kW); a CDU must be rated above the sum of server coolant loads plus margin, and coolant flow rate must meet each node's specified liters-per-minute.

Network cabling density: a rail-optimized GPU fabric can require hundreds of fiber/copper links per rack; cable management and bend-radius limits affect signal integrity at 400G/800G.

Environmental monitoring: track inlet/outlet temperature, humidity, differential pressure, and leak sensors; humidity below recommended range increases ESD risk, above range increases condensation risk on cold plates.