NCA Foundations cheat sheet

GPU vs CPU design: CPUs dedicate large caches and control logic to few latency-optimized cores; GPUs dedicate most transistors to thousands of throughput-oriented ALU lanes.

Amdahl's law: speedup = 1 / ((1 - p) + p/s), where p is the parallel fraction and s the parallel speedup; serial fractions cap achievable gain.

Gustafson's law scales problem size with processor count, so observed speedup grows when work per processor is held constant rather than total work.

A GPU is a discrete accelerator on PCIe or an NVLink-connected module; host memory is not directly dereferenceable by device code without unified or pinned memory.

Roofline model: attainable performance = min(peak FLOP/s, arithmetic intensity x memory bandwidth); arithmetic intensity is FLOPs per byte moved.

Key platform components: GPU, driver, CUDA toolkit, cuDNN/TensorRT libraries, and orchestration such as NGC containers and MIG partitioning.

Bandwidth hierarchy for planning: HBM device memory is far slower than on-chip SRAM, and PCIe host transfers are slower still than device HBM.

Multi-GPU topologies (NVLink, NVSwitch, PCIe) set the communication ceiling; all-reduce cost scales with model size and inverse link bandwidth.

SIMT execution: threads run in warps of 32 on Volta-and-later architectures; divergence within a warp serializes the divergent paths.

Latency hiding relies on occupancy: many resident warps per SM let the scheduler issue other warps while some stall on memory or dependencies.

Memory spaces: registers (per thread, fastest), shared memory (per block, explicitly managed), L1/L2 cache, global memory (HBM), plus constant and texture paths.

Coalescing rule: adjacent threads should access adjacent addresses; strided or misaligned access multiplies the number of memory transactions.

Synchronization: __syncthreads() is a block-wide barrier; inter-block ordering requires separate kernels, atomics, or cooperative groups.

Occupancy limiters include registers per thread, shared memory per block, block size, and architectural warp slots per SM.

Shared memory bank conflicts occur when threads in a warp hit the same bank on different addresses; padding arrays by one element often removes them.

Streams and events allow overlap of kernel execution with host-device copies; asynchronous copies require pinned host memory for best throughput.

Stack layers, bottom up: GPU hardware, driver (with CUDA driver API), CUDA runtime and toolkit, accelerated libraries, then frameworks and applications.

CUDA toolkit supplies nvcc, the runtime API, profiling tools (Nsight Systems, Nsight Compute), and libraries such as cuBLAS, cuFFT, cuSPARSE, and NCCL.

cuDNN provides tuned deep learning primitives (convolutions, pooling, normalization); TensorRT builds and optimizes inference engines for deployment.

CUDA compatibility: the driver must be at least as new as the toolkit's minimum requirement; forward compatibility packages relax this for some data-center GPUs.

Compute capability (e.g., 7.0, 8.0, 9.0) determines available instructions and features; code compiled for a higher capability will not run on older hardware.

NGC containers bundle CUDA, cuDNN, frameworks, and NCCL at tested versions, reducing dependency drift across nodes.

Programming models beyond CUDA C++ include CUDA Python (Numba, CuPy), OpenACC directives, and framework-level kernel generation.

NCCL handles collective communication (all-reduce, broadcast, all-gather) across GPUs and nodes, choosing rings or trees based on topology.

Workflow stages: data collection and labeling, preprocessing, model design, training, evaluation, hyperparameter tuning, export, and deployment.

Training loop: forward pass, loss computation, backward pass via automatic differentiation, gradient reduction across replicas, then optimizer step.

Precision options: FP32 baseline, TF32 for tensor-core matmuls, FP16/BF16 mixed precision with loss scaling, and FP8 on newer architectures.

Data pipeline must keep GPUs fed; input loading, decoding, and augmentation on CPU can bottleneck training if not parallelized or offloaded.

Batch size interacts with learning rate; large-batch training often needs learning-rate warmup and scaling rules to remain stable.

Overfitting signals: training loss falls while validation loss rises; remedies include augmentation, regularization, and early stopping.

Distributed strategies: data parallelism replicates the model, tensor parallelism splits layers, pipeline parallelism splits depth; large models combine them.

Checkpointing stores weights and optimizer state; frequency trades off lost work on failure against I/O overhead and storage cost.

RAPIDS libraries (cuDF, cuML, cuGraph, cuSpatial) mirror pandas, scikit-learn, and NetworkX APIs while executing on GPUs.

Dask-cuDF partitions a dataframe across GPUs; the partition count and shuffle strategy dominate performance for group-by and join operations.

Zero-copy interchange protocols (Apache Arrow) avoid serialization between GPU dataframe and other tools; CPU-GPU transfer remains the common cost.

cuML estimators follow fit/predict semantics; single-GPU algorithms may not scale to datasets exceeding device memory without partitioning.

RAPIDS Accelerator for Apache Spark replaces CPU Spark SQL and DataFrame operators with GPU equivalents when types and operations are supported.

UDFs and Python row-wise functions usually force fallback to CPU; vectorized expressions keep execution on the GPU.

Feature engineering on GPU benefits most when the full pipeline stays on device; round-tripping to host per step erases the gain.

Profiling data workloads: measure I/O, decompression, transfer, and compute separately, since decompression of columnar formats is often the bottleneck.

Inference optimization levers: reduced precision (FP16, INT8, FP8), kernel fusion, layer and tensor elimination, and dynamic batching.

TensorRT workflow: build a network definition or parse ONNX, select precision and tactics, then serialize an engine tied to GPU architecture and TensorRT version.

Quantization approaches: post-training quantization with calibration data, quantization-aware training, and smooth-quant-style activation rescaling.

Structured pruning removes whole channels or heads and yields real speedups; unstructured sparsity needs hardware support to convert to latency gains.

Knowledge distillation trains a smaller student on a larger teacher's outputs, trading accuracy for lower latency and memory.

Serving metrics to watch: time to first token, inter-token latency, throughput in requests or tokens per second, and queue depth under load.

Batching trade-off: larger batches raise throughput but increase per-request latency; continuous batching mitigates this for variable-length generation.

Engine portability: a serialized TensorRT engine is not portable across GPU architectures or major TensorRT versions and must be rebuilt.

nvidia-smi reports utilization, memory, temperature, power, ECC errors, and running processes; query mode supports scripted monitoring.

DCGM (Data Center GPU Manager) provides health checks, field-level metrics, and integration with Prometheus and Kubernetes GPU dashboards.

Xid errors are driver-reported GPU faults; common causes include fallen-off-bus, ECC, and illegal memory access, and codes should be looked up before action.

Out-of-memory errors: distinguish device allocation failure from host pinned-memory exhaustion; check batch size, leaks, and fragmentation.

Thermal and power throttling show as clocks below boost with high temperature; airflow, fan curves, and power caps are the usual levers.

NCCL hangs often trace to mismatched collective calls, network fabric issues, or missing GPU visibility; set NCCL debug logging to localize.

Container GPU access requires the NVIDIA container toolkit and a device request; missing driver mounts make CUDA appear absent inside the container.

Driver and toolkit mismatch surfaces as initialization errors; verify driver version against the CUDA version the toolkit requires.

Product families: GeForce for workstations and consumers, RTX for professional visualization, data-center GPUs (A100, H100, L40S class) for training and inference.

Supercomputing platforms combine GPUs with high-speed interconnect (NVLink, InfiniBand) for large-scale training; Grace-based systems pair CPU and GPU memory.

Edge and embedded offerings (Jetson) target robotics and vision with constrained power; DRIVE targets automotive autonomy.

Industry use cases: conversational AI, recommender systems, computer vision inspection, drug discovery simulation, and climate and financial modeling.

Professional practice: pin library versions, record GPU model and driver in experiment metadata, and keep reproducible container images.

Responsible AI practice includes dataset documentation, bias evaluation, license compliance for model weights, and data privacy review before training.

Cost and capacity planning compares on-premises GPU utilization against cloud burst; utilization rate is the key driver of unit economics.

Collaboration norms: share benchmarks with hardware and software versions, use profiling evidence for optimization claims, and cite measured baselines.