NVIDIA NCP-AIN cheat sheet

InfiniBand (IB) is a lossless, credit-based fabric: link-level flow control uses credits so packets are never dropped; congestion control is handled by the subnet manager, not by TCP-style loss recovery.

RoCEv2 encapsulates IB transport over UDP/IP (default port 4791); it is routable but requires PFC (priority flow control) and ECN/DCQCN to approximate IB losslessness.

NVIDIA Quantum InfiniBand HDR = 200 Gb/s per port, NDR = 400 Gb/s, XDR = 800 Gb/s; Spectrum Ethernet covers 100/200/400 Gb/s (SN4000/SN5000 families).

NVIDIA SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) performs all-reduce and similar collectives inside Quantum switches, cutting host round-trips and CPU load.

A subnet manager (OpenSM or UFM) discovers topology, assigns LIDs, programs forwarding tables, and configures QoS; one SM is master, others standby.

GPUDirect RDMA lets a NIC DMA directly to GPU memory over PCIe, bypassing host CPU copies; requires IOMMU/pinning configuration and matching driver stack.

Fat-tree/rail-optimized topologies: rail-aligned cabling lets each GPU on a node use a dedicated NIC to the same rail, so all-reduce traffic stays on one rail per step.

MTU and buffer sizing: IB MTU options are 256/512/1024/2048/4096 bytes; jumbo Ethernet frames (9000) reduce per-packet overhead for storage and east-west traffic.

Diagnostic rule: rising symbol errors, link flaps, or "PortXmitDiscards" on an IB port point to cable/transceiver faults before switch configuration.

Air cooling guidance: ASHRAE TC 9.9 recommends inlet temperatures of 18–27 °C (64.4–80.6 °F) for Class A1/A2 equipment; exceeding it raises fan power and throttling risk.

Power budget formula: rack kW = sum of device nameplate watts × utilization factor; PUE = total facility power ÷ IT equipment power, so lower PUE means better efficiency.

GPU server density: an 8-GPU HGX/DGX-class node commonly draws 6–10 kW, forcing liquid cooling (direct-to-chip cold plates) or rear-door heat exchangers in dense racks.

Redundancy notation: N+1 means one spare unit; 2N means fully duplicated paths; A/B feed pairs and dual-corded PDUs are standard for AI clusters needing maintenance without downtime.

NVIDIA DGX SuperPOD reference architecture pairs compute nodes with Quantum InfiniBand leaf/spine switches, UFM management, and a validated cabling plan.

Cabling: DAC for short in-rack runs, AOC for mid distances, optical transceivers (QSFP/OSFP) for long runs; bend radius and airflow blockage are common field failures.

Commissioning sequence: physical install → firmware/driver baseline → fabric cabling validation → SM/QoS configuration → burn-in (NCCL all-reduce, GPU stress) → acceptance.

Monitoring signals: GPU temperature, ECC/xid errors, NVLink error counters, switch symbol errors, PSU load, and inlet/outlet delta-T, all fed to DCGM and a fabric manager.

CUDA hierarchy: threads → warps (32 threads) → blocks → grids; a warp executes one instruction stream, so divergence within a warp serializes execution.

Memory hierarchy: registers and shared memory are on-chip and fastest; global (HBM) memory is large but high latency; coalesced global accesses maximize DRAM efficiency.

Mixed precision: FP16/BF16 tensor cores deliver far higher throughput than FP32; BF16 keeps FP32's exponent range, making it the default for large-model training.

Parallelism choices: data parallel (replicate model, split batch), tensor parallel (split layers within a node, NVLink-friendly), pipeline parallel (split layers across nodes), plus ZeRO/FSDP sharding.

Scaling rule: strong scaling keeps problem size fixed as GPUs grow; weak scaling grows the problem with the hardware. Communication overhead limits strong scaling.

NCCL implements collectives (all-reduce, all-gather, broadcast, reduce-scatter) and picks ring or tree algorithms based on topology and message size.

Transformer compute is dominated by GEMMs; attention cost scales with sequence length squared, which motivates FlashAttention-style tiling and KV-cache management.

Inference optimization: quantization (FP8/INT8/INT4), batching, KV-cache reuse, and TensorRT/TensorRT-LLM engine builds trade accuracy and latency against throughput.

NVIDIA software stack: driver → CUDA toolkit → cuDNN/cuBLAS/NCCL → frameworks (PyTorch, TensorFlow) → deployment (Triton Inference Server, NIM).

Cloud GPU instance families map to GPU generations: NVIDIA A100 for general training/inference, H100 for large-model training, L4/A10 for inference and graphics, T4 for light inference.

NVIDIA AI Enterprise is a licensed software suite (CUDA, Triton, TensorRT, NIM, vGPU) with support; it is required for certified deployment on vSphere/cloud and is licensed per GPU or per core.

NVIDIA vGPU lets one physical GPU be partitioned into virtual GPUs with fixed framebuffer; MIG instead partitions an Ampere+ GPU into isolated instances with dedicated memory slices and SMs.

Cloud networking for AI: provider offerings such as Elastic Fabric Adapter or RDMA-enabled interconnects are needed for multi-node training; ordinary TCP VPC networking bottlenecks all-reduce.

Cost model: on-demand vs reserved/committed-use vs spot; spot instances suit checkpointed training, while inference serving favors reserved capacity for latency stability.

Storage tiers: local NVMe for dataset caching, parallel file systems (Lustre, GPFS, cloud FSx-type) for shared training data; throughput must keep GPUs fed.

NVIDIA NGC and NIM provide containerized frameworks and optimized inference microservices; containers pin CUDA/driver compatibility, so host driver versions must satisfy the container's requirements.

Deployment decision rule: single node → NVLink + NCCL; multi-node → InfiniBand/RDMA fabric; latency-sensitive inference → edge or regional region close to users.

InfiniBand link speeds: SDR 10, DDR 20, QDR 40, FDR 56, EDR 100, HDR 200, NDR 400, XDR 800 Gb/s per port (signaling rate × lanes, encoded).

IB uses 16-bit LIDs for local routing and 128-bit GIDs (IPv6-format) for global addressing; the SM assigns LIDs during discovery.

RoCEv2 runs over UDP port 4791 and needs lossless Ethernet: PFC on a priority class plus ECN marking with DCQCN congestion control.

NVIDIA Spectrum switches run Cumulus Linux or SONiC; Quantum switches run MLNX-OS and are managed by UFM or OpenSM.

NVLink is the intra-node GPU interconnect (NVLink 4 on H100: 900 GB/s bidirectional per GPU); it is separate from the InfiniBand fabric.

NCCL topology detection uses NVML and fabric info to choose rings/trees; NCCL_IB_HCA and NCCL_SOCKET_IFNAME control which interfaces are used.

MTU mismatch or PFC misconfiguration on RoCE typically shows as retransmits and latency spikes under load, not as total link failure.

PUE = total facility energy ÷ IT energy; 1.0 is ideal, typical efficient facilities are 1.1–1.4.

ASHRAE A1 inlet range is 18–27 °C; liquid cooling (direct-to-chip) becomes necessary above roughly 10–15 kW per rack for GPU nodes.

Power redundancy: 2N = fully duplicated; N+1 = one spare; dual-corded servers connect to A and B PDUs on separate feeds.

DGX SuperPOD uses a rail-optimized InfiniBand fabric with leaf/spine switches and UFM for management.

Cable media: DAC (short, in-rack), AOC (mid), optical transceivers with fiber (long, inter-row).

DCGM monitors GPU health: temperature, ECC errors, xid errors, power, and utilization; fabric managers monitor switch symbol errors.

Burn-in acceptance typically runs NCCL all-reduce benchmarks plus GPU stress to validate fabric and thermal behavior before production.

CUDA warp = 32 threads; divergence within a warp serializes the divergent paths.

On-chip memory (registers, shared memory) is far faster than HBM global memory; coalesced global loads maximize bandwidth.

BF16 keeps FP32 exponent range and is preferred for large-model training; FP16 has more mantissa but narrower range.

Parallelism: data parallel replicates the model; tensor parallel splits layers within a node (NVLink); pipeline parallel splits layers across nodes.

NCCL provides all-reduce, all-gather, reduce-scatter, and broadcast, selecting ring or tree algorithms by topology and message size.

Transformer attention cost scales with sequence length squared; FlashAttention tiles it to reduce memory traffic.

Inference optimization levers: quantization (FP8/INT8/INT4), batching, KV-cache reuse, and TensorRT/TensorRT-LLM engines.

Stack order: driver → CUDA → cuDNN/cuBLAS/NCCL → framework (PyTorch) → serving (Triton, NIM).

NVIDIA AI Enterprise is a licensed, supported software suite (CUDA, Triton, TensorRT, NIM, vGPU) certified on vSphere and major clouds.

MIG partitions an Ampere+ GPU into isolated instances with dedicated memory and SMs; vGPU time-slices a GPU with a fixed framebuffer per virtual GPU.

Multi-node cloud training requires RDMA-capable networking (EFA-style or InfiniBand); plain TCP VPC networking bottlenecks all-reduce.

Purchasing models: on-demand (flexible), reserved/committed (cheaper, stable), spot (cheap, preemptible) - spot fits checkpointed training.

NVIDIA NGC hosts containerized frameworks; container CUDA versions must be compatible with the host driver.

Storage tiers: local NVMe cache, parallel file systems for shared datasets; throughput must sustain GPU feeding.

NIM packages optimized inference microservices as containers with standardized APIs.

InfiniBand is credit-based and lossless at link level; congestion is managed by the SM and congestion control, not by packet drops.

Quantum InfiniBand speeds: HDR 200, NDR 400, XDR 800 Gb/s per port.

SHARP performs in-network aggregation (all-reduce) inside Quantum switches, reducing host traffic.

One subnet manager is master, others standby; it assigns LIDs and programs forwarding tables.

GPUDirect RDMA allows NIC-to-GPU direct DMA over PCIe, bypassing host memory copies.

Rail-optimized cabling aligns each GPU's NIC to the same rail switch, keeping all-reduce traffic on one rail per step.

RoCEv2 uses UDP port 4791; requires PFC + ECN/DCQCN to avoid drops under congestion.

PUE = total facility power ÷ IT power; lower is better, efficient facilities ~1.1–1.4.

ASHRAE A1 inlet temperature range: 18–27 °C.

8-GPU HGX/DGX nodes commonly draw 6–10 kW, requiring liquid cooling or rear-door heat exchangers at density.

Redundancy: 2N fully duplicated, N+1 one spare; dual-corded servers on separate A/B PDUs.

Cabling media: DAC short in-rack, AOC mid, optical fiber long runs.

Commissioning order: install → firmware baseline → cabling validation → SM/QoS config → burn-in → acceptance.

DCGM tracks GPU temperature, ECC/xid errors, power, and utilization for operations.

CUDA execution: threads → warps (32) → blocks → grids; warp divergence serializes.

Registers and shared memory are on-chip and fast; HBM global memory is large but high latency.

BF16 has FP32's exponent range; FP16 has more mantissa bits but narrower range.

Data parallel replicates the model; tensor parallel splits layers within a node; pipeline parallel splits across nodes.

Strong scaling fixes problem size; weak scaling grows the problem with hardware.

NCCL picks ring or tree algorithms based on topology and message size.

Quantization (FP8/INT8/INT4) and KV-cache reuse are primary inference throughput levers.

NVIDIA AI Enterprise is a licensed supported software suite certified on vSphere and major clouds.

MIG gives isolated GPU instances with dedicated memory/SMs; vGPU time-slices with fixed framebuffer.

Multi-node cloud training needs RDMA-capable networking; TCP VPC networking bottlenecks all-reduce.

Spot instances suit checkpointed training; reserved capacity suits latency-stable inference.

NGC containers must match host driver/CUDA compatibility.

Parallel file systems (Lustre, GPFS, cloud equivalents) supply shared training data throughput.

NIM provides containerized optimized inference microservices with standardized APIs.