NVIDIA NCA-AIIO cheat sheet

AI ⊃ ML ⊃ DL: machine learning learns patterns from data instead of explicit rules; deep learning stacks many neural-network layers so features are learned automatically.

Training runs a forward pass plus a backward pass that computes gradients and updates weights; inference runs only the forward pass and is optimized for latency and throughput.

Classification metrics: Precision = TP/(TP+FP), Recall = TP/(TP+FN), F1 = 2PR/(P+R); F1 is the harmonic mean and balances false positives against false negatives.

Weight memory estimate: bytes ≈ parameter count × bytes per parameter - FP32 = 4, FP16/BF16 = 2, FP8/INT8 = 1; total GPU memory also needs activations, gradients, and optimizer state.

Transformer self-attention computes softmax(QKᵀ/√d_k)·V, so compute and memory grow roughly with the square of sequence length (O(n²)).

An epoch is one full pass over the training set; steps per epoch ≈ dataset size ÷ batch size, so doubling batch size halves the steps per epoch.

Data parallelism replicates the whole model per GPU and splits the batch, all-reducing gradients; tensor and pipeline parallelism shard the model itself across GPUs.

Reinforcement learning trains an agent to maximize cumulative reward; RLHF aligns language models by optimizing against a reward model built from human preference data.

Quantization lowers numeric precision (FP16 → FP8/INT8) to shrink memory footprint and increase bandwidth efficiency, trading a small amount of accuracy for speed.

A Streaming Multiprocessor (SM) is the GPU's core execution block: it holds CUDA cores, Tensor Cores, registers, and shared memory, and thread blocks are scheduled onto SMs.

GPU memory hierarchy is registers → shared memory/L1 → L2 → HBM; HBM bandwidth (about 3.35 TB/s on H100 SXM) frequently limits large-model inference more than capacity does.

Fourth-generation NVLink on H100 provides about 900 GB/s bidirectional per GPU, and NVSwitch extends that full-bandwidth all-to-all connectivity across a node's GPUs.

PCIe Gen4 x16 delivers about 32 GB/s per direction and Gen5 doubles it to about 64 GB/s; both are far below NVLink, so peer GPU traffic is kept on NVLink.

MIG (Multi-Instance GPU) splits one physical GPU into up to seven isolated instances with dedicated memory slices, cache, and SMs for multi-tenant QoS.

GPUDirect RDMA lets network adapters read and write GPU memory directly across the fabric, eliminating host-memory bounce buffers and reducing inter-node latency.

Scale-out fabrics use InfiniBand or RoCEv2 over Ethernet with RDMA; fat-tree and rail-optimized topologies are common, and oversubscription above 1:1 raises congestion risk.

PUE = total facility power ÷ IT equipment power; 1.0 is the theoretical ideal, and cooling choice (air vs liquid) plus power delivery set achievable rack density.

Storage is tiered: local NVMe for scratch and fast checkpoints, parallel file systems (Lustre, GPFS) for shared high-throughput reads, and object storage for durable datasets.

nvidia-smi reports point-in-time GPU inventory, utilization, memory, temperature, power, and processes; DCGM plus DCGM Exporter, Prometheus, and Grafana provide continuous telemetry and alerting.

Core GPU health signals are XID events, ECC single- and double-bit errors, retired pages, temperature, power draw, throttle reasons, and NVLink link status.

XID errors are driver-reported GPU events in the kernel log; codes such as 79 (GPU has fallen off the bus) typically require node drain, reset, or RMA rather than a software fix.

Kubernetes schedules GPUs through the NVIDIA GPU Operator and device plugin; MIG-backed workloads require MIG-aware device advertisement and a matching MIG strategy.

Availability = MTBF ÷ (MTBF + MTTR); raising mean time between failures or shortening mean time to repair both increase the usable fraction of time.

Checkpoint period ≈ √(2 × checkpoint cost × MTBF) balances the overhead of writing checkpoints against the work lost when a job is interrupted.

CUDA compatibility rule: the installed driver must support the CUDA toolkit version used by the workload; the NVIDIA Container Toolkit injects the host driver libraries into containers.

Gang scheduling admits all parts of a multi-node job together or not at all, preventing partial allocations that deadlock and leave reserved GPUs idle.

GPU allocation counts devices reserved by jobs while GPU utilization measures actual compute activity; high allocation with low utilization indicates idle reserved capacity or poor job packing.

Driver and firmware changes are rolled out through canary nodes and validated with burn-in tests and diagnostics before nodes are returned to the production pool.