300-640 DCAI cheat sheet

Reference AI cluster tiers: management/control plane, compute (GPU) plane, storage plane, and high-speed data plane are separated by design so control traffic never contends with collective traffic.

Rack power density rule: a rack of 8-way HGX-class GPU servers at ~10 kW per server exceeds 70 kW per rack, which forces liquid cooling or rear-door heat exchangers rather than air-only design.

Fat-tree (Clos) with full bisection bandwidth is the default fabric topology for training; oversubscription above 1:1 is acceptable only for inference or storage-facing tiers.

Design for east-west dominance: in distributed training, inter-GPU traffic exceeds north-south client traffic by orders of magnitude, so spine capacity is sized from collective volume, not user demand.

GPU-to-CPU ratio guideline: training nodes typically pair 1 CPU socket per 4-8 GPUs, while inference nodes may run closer to 1:1 depending on token throughput and preprocessing.

Non-blocking fabric formula: required spine uplinks = (leaf downlinks x leaf count) / spine count; verify the result equals total leaf downlink bandwidth for 1:1 oversubscription.

Power and cooling budget: sum GPU TDP, CPU TDP, NIC and optics, plus 20-30% headroom for fans and PSU losses; facility capacity is usually the binding constraint, not server count.

Separate the AI zone from the enterprise zone with a dedicated border (firewall plus proxy) so model traffic, dataset transfer, and management access follow distinct policy paths.

NVLink/NVSwitch provides GPU-to-GPU bandwidth roughly an order of magnitude above PCIe Gen5 x16 (~64 GB/s per direction), so intra-node collectives should stay on NVLink and never traverse the NIC.

GPUDirect RDMA lets a NIC write directly into GPU memory, bypassing host memory copies; it requires IOMMU/ACS configuration and peer-to-peer support on the PCIe root complex.

MIG partitions a single GPU into isolated slices with dedicated memory and SMs; it suits inference multi-tenancy but reduces the largest single-job GPU memory footprint.

Collective algorithms: ring all-reduce moves ~2(N-1)/N times the message size per rank, so bandwidth cost is nearly independent of GPU count, while latency scales with ring depth.

Gradient accumulation trades memory for time: effective batch = micro-batch x data-parallel ranks x accumulation steps; it does not change total FLOPs.

Mixed precision: BF16 has the same exponent range as FP32 and avoids loss scaling; FP16 usually needs dynamic loss scaling to prevent gradient underflow.

GPU memory planning: parameters + gradients + optimizer states (Adam holds two extra FP32 tensors) + activations; ZeRO/FSDP shards optimizer state, gradients, then parameters across ranks.

PCIe topology check: confirm GPUs and NICs sit under the same CPU root complex where peer-to-peer is required; cross-socket traffic adds latency and can disable P2P.

Parallel file systems (Lustre, GPFS/Storage Scale, WEKA) are the standard for training datasets; NFS is acceptable for small home directories but not for checkpoint bursts.

Checkpoint sizing formula: checkpoint bytes = parameters x bytes per element x replication factor; a 70B model in BF16 with optimizer state can exceed 1 TB per save, so write bandwidth must be sized for burst.

Storage tiering rule: hot tier (NVMe parallel FS) for active datasets and checkpoints, warm tier (object store) for curated datasets and model registry, cold/archive for raw ingestion.

Object storage (S3-compatible) is the landing zone for datasets and model artifacts; parallel FS is the working set; avoid running training directly against object storage unless a caching layer exists.

Throughput sizing: aggregate read bandwidth should exceed the number of training nodes x per-node NIC bandwidth to prevent I/O from becoming the collective bottleneck.

Data versioning and lineage: every training run should reference an immutable dataset snapshot plus a model registry entry so results are reproducible and auditable.

Small-file problem: millions of tiny files degrade metadata servers; pack datasets into shards (e.g., WebDataset/TFRecord) to reduce metadata operations.

Snapshot and backup policy: checkpoints are not backups; keep versioned copies in a separate fault domain and test restore time against the training restart window.

RoCEv2 requires lossless Ethernet: enable PFC on priority traffic and ECN marking with DCQCN or a comparable congestion control scheme to avoid throughput collapse.

InfiniBand NDR is 400 Gb/s per port and XDR is 800 Gb/s; Ethernet equivalents are 400G and 800G, so port-count math must state the generation explicitly.

Rail-optimized design: each GPU index connects to the same leaf rail across nodes, so a collective between matching GPU indices stays within one leaf and one hop.

Adaptive routing (per-packet spraying in IB, or flowlet/ECMP with hash entropy in Ethernet) reduces hash collisions on elephant flows typical of all-reduce.

Bandwidth-delay product matters: at 400G and ~1 microsecond latency, in-flight bytes are modest, but incast from synchronized collectives still requires deep buffers or ECN.

Optics budget: 400G DR4 reaches ~500 m over single-mode, 400G FR4 ~2 km; DAC/AEC covers in-rack and short row distances at lower cost and power.

Cable and transceiver inventory should be planned per port pair; a 1:1 non-blocking fabric with N leaves of 64 ports needs N x 32 spine-facing links plus optics on both ends.

Telemetry: collect per-port counters, PFC pause frames, ECN marks, and buffer utilization; rising PFC pauses are the earliest signal of congestion before job slowdowns appear.

Slurm is the dominant HPC/AI scheduler; Kubernetes with device plugins and the NVIDIA GPU Operator is the dominant container platform; many sites run Slurm on bare metal and Kubernetes for inference services.

GPU scheduling rule: request whole GPUs via device plugin resources, use node affinity/taints to pin jobs to the correct GPU generation, and avoid bin-packing that fragments MIG slices.

Infrastructure as code: define fabric, compute, and storage declaratively (Terraform/Ansible) and store the desired state in version control so rebuilds are reproducible.

Job health checks: validate GPU ECC errors, XID codes, NVLink status, and fabric reachability before allocating a node; a single degraded GPU can stall a large collective.

Observability stack: node exporter plus DCGM exporter for GPU metrics, fabric telemetry, and scheduler queue depth; alert on job start latency and GPU idle time, not just utilization.

Rolling upgrades: drain nodes via the scheduler, cordon in Kubernetes, verify collective health on a canary job, then return capacity; never upgrade fabric and compute simultaneously.

Runbook automation: standardize node burn-in, driver/CUDA/firmware baselines, and image builds so a replacement node reaches production state without manual steps.

Chargeback and quotas: map scheduler accounts to projects, enforce GPU-hour quotas, and report utilization per team to prevent idle reservations from stranding capacity.

Multi-tenancy isolation: separate namespaces/partitions, enforce network policy between tenants, and never share a GPU across trust boundaries without MIG or time-slicing controls.

Model and data protection: encrypt datasets and checkpoints at rest, use short-lived credentials for object storage access, and log every model artifact pull in the registry.

Supply chain: pin container images by digest, scan for CVEs, verify model weights from external sources, and treat serialized model formats as executable code.

Resiliency pattern: checkpoint frequently, detect node failure via scheduler and fabric telemetry, and restart from the last checkpoint; MTBF of large jobs falls as GPU count rises.

Redundancy rules: dual power feeds per rack, redundant leaf-spine paths, and N+1 cooling; a single PSU or leaf failure must not fail an entire training job.

Sustainability metrics: track PUE, GPU utilization, and tokens or samples per kilowatt-hour; idle reserved GPUs are the largest avoidable efficiency loss.

Cooling choice: liquid cooling (direct-to-chip) enables higher rack density and lower PUE than air; waste-heat reuse is viable where facility plumbing exists.

Least privilege for AI ops: separate roles for fabric admins, cluster admins, and data scientists; audit privileged commands and rotate API tokens used by automation.