NCA Foundations key terms

A processor built from many small cores that run thousands of threads at once, designed for parallel arithmetic. It works alongside a CPU, which handles control flow and orchestration.

One of the many scalar arithmetic units inside an NVIDIA streaming multiprocessor. Each core executes a thread's instruction, and large groups of cores deliver the platform's parallel throughput.

A self-contained scheduling and execution block within an NVIDIA GPU that contains CUDA cores, tensor cores, registers, shared memory, and warp schedulers. A GPU contains many SMs operating concurrently.

A specialized execution unit that performs small matrix multiply-accumulate operations in a single instruction, greatly accelerating deep learning math. Tensor cores support reduced precision such as FP16, BF16, and FP8.

Single Instruction, Multiple Threads: the GPU execution model in which one instruction is issued for a group of threads, each with its own data and register state. Divergent branches within a group serialize.

A fixed group of 32 threads that a GPU schedules together as a unit. Warps are the granularity for scheduling, and each thread in a warp has a unique lane identifier.

A condition where threads in a warp take different branch paths, forcing the paths to execute one after another and reducing efficiency. It is often avoided by restructuring conditionals or data layout.

A function written for the GPU that is launched to run across many threads in parallel. Kernels are the basic unit of work submitted to a device.

The complete set of thread blocks created by one kernel launch. Its size and shape are chosen by the programmer to cover the data being processed.

A group of threads that execute together on one streaming multiprocessor and can cooperate through shared memory and synchronization. Block size strongly affects occupancy and performance.

A small, fast, programmer-managed memory space on each streaming multiprocessor shared by the threads of a block. It is used to reuse data and reduce traffic to global memory.

The hardware combining of memory requests from threads in a warp into fewer, larger transactions when the threads access adjacent addresses. Poor coalescing wastes bandwidth and slows kernels.

The ratio of active warps on a streaming multiprocessor to the maximum it supports. Higher occupancy can hide memory latency, but it is limited by registers, shared memory, and block size.

The host is the CPU and its memory that runs the operating system and launches work; the device is the GPU and its memory that executes kernels. Data moves between them over a bus such as PCIe.

NVIDIA's parallel computing platform and programming model that lets developers write GPU-accelerated code in languages such as C++, Python, and Fortran. It includes compilers, libraries, and runtime APIs.

The collection of compilers, libraries, debuggers, and profiling tools used to build and tune GPU applications. It includes the nvcc compiler, the CUDA runtime, and developer utilities.

NVIDIA's GPU-accelerated implementation of the Basic Linear Algebra Subprograms, providing optimized routines such as matrix multiplication. Many deep learning frameworks rely on it for dense linear algebra.

NVIDIA's library of tuned primitives for deep neural networks, covering convolution, pooling, normalization, and activation operations. Frameworks call it to obtain fast, hardware-specific implementations.

NVIDIA's library for collective communication across multiple GPUs and nodes, supporting operations such as all-reduce and broadcast. It underpins efficient multi-GPU and distributed training.

An inference optimizer and runtime that fuses layers, selects precision, and picks fast kernels for a trained model. It produces an optimized engine targeted to specific GPU hardware.

Running a trained model on new inputs to produce predictions, as opposed to training it. Inference workloads emphasize low latency, high throughput, and efficient memory use.

Reducing the numerical precision of model weights or activations, for example from FP32 to FP16 or INT8, to shrink memory use and speed up computation with limited accuracy loss.

Training or inference that combines lower-precision arithmetic, such as FP16 or BF16, with FP32 where numerical stability matters. It speeds up math on tensor cores while preserving accuracy.

Removing weights, neurons, or structures from a trained network that contribute little to output quality. The result is a smaller, faster model that uses less memory.

Grouping multiple input samples into one forward pass so the GPU can process them together. Larger batches raise throughput but also raise latency and memory use.

Throughput is how much work completes per unit time, such as inferences per second; latency is the time one request takes. Optimizing one often trades against the other.

A training approach that learns a mapping from inputs to labels using a dataset of labeled examples. The model is optimized to minimize error on those examples and generalize to new data.

Training adjusts model parameters by repeatedly computing gradients and updating weights; inference uses the frozen trained model to make predictions. They have different compute and memory profiles.

The algorithm that computes gradients of a loss with respect to each model parameter by applying the chain rule backward through the network. Optimizers then use these gradients to update weights.

An epoch is one full pass over the training dataset; batch size is the number of samples processed before one weight update. Together they set how many updates occur and how much memory is used.

Splitting model training across multiple GPUs or nodes to reduce wall-clock time or fit larger models. Common strategies are data parallelism, tensor parallelism, and pipeline parallelism.

NVIDIA's suite of open-source GPU-accelerated data science libraries, including cuDF for dataframes and cuML for machine learning. It lets familiar Python workflows run on GPUs with minimal code change.

A GPU dataframe library in the RAPIDS suite that provides pandas-like APIs for loading, filtering, joining, and aggregating tabular data on the GPU. It accelerates data preparation for analytics and ML.

A language-independent columnar memory format for flat and hierarchical data. It enables zero-copy data exchange between tools and is the in-memory foundation for many GPU dataframe libraries.

A high-bandwidth direct interconnect between GPUs, and in some systems between GPU and CPU, that is faster than PCIe. It speeds up multi-GPU communication for training and large-model inference.

A CUDA feature that provides a single address space shared by CPU and GPU, with the system migrating pages on demand. It simplifies programming but can incur migration overhead if access patterns are poor.

A system-wide performance analysis tool that traces CPU and GPU activity on a timeline. It helps identify bottlenecks such as idle GPUs, slow data transfers, and synchronization stalls.

A kernel-level profiler that reports detailed metrics such as memory throughput, occupancy, and instruction mix for individual GPU kernels. It guides low-level optimization work.

A command-line utility for querying and managing NVIDIA GPUs, reporting utilization, memory use, temperature, and running processes. It is a first tool for checking device health and load.

Dedicated high-bandwidth memory attached to the GPU that holds model weights, activations, and data during computation. Running out of it causes out-of-memory errors that stop jobs.