NVIDIA NCP-AIN key terms

A high-throughput, low-latency switched fabric used in GPU clusters, supporting Remote Direct Memory Access and hardware-based congestion control for scaling distributed training workloads across many nodes.

Remote Direct Memory Access over Converged Ethernet, which carries RDMA traffic on Ethernet networks. Version 2 adds IP routability, and lossless configuration via PFC and ECN is required to preserve performance.

Remote Direct Memory Access lets one machine read or write another machine's memory without involving either CPU, cutting latency and CPU overhead. It underpins efficient collective communication in distributed GPU training.

Priority Flow Control is an Ethernet link-level mechanism that pauses specific traffic classes instead of the whole link, preventing buffer overflow for lossless RDMA flows while allowing other priorities to continue.

Explicit Congestion Notification marks packets in switches before buffers fill, letting endpoints reduce sending rate rather than dropping traffic. It is a preferred congestion signal for lossless RoCE fabrics.

A hierarchical spine-leaf network design in which every leaf switch reaches every spine with equal hop count, providing predictable latency and full bisection bandwidth for large GPU clusters.

A two-tier data center network where leaf switches connect servers and spine switches interconnect leaves, enabling east-west traffic to traverse a consistent number of hops with high aggregate throughput.

The total bandwidth available across a network's midpoint when divided into two equal halves. It bounds how quickly distributed training jobs can exchange gradients and is a key cluster design metric.

Data center traffic that flows between servers rather than to or from external clients. Distributed AI training generates heavy east-west traffic during gradient exchange, shaping network capacity requirements.

Traffic entering or leaving the data center, typically between clients and internal services. It is distinct from server-to-server flows and is handled by edge routers and load balancers.

Algorithms and hardware mechanisms that detect network saturation and adjust sending rates or routing to avoid packet loss and preserve low latency, especially important for collective AI communication patterns.

A switching capability that dynamically selects paths per packet or flow based on current fabric load, reducing hot spots and improving utilization in large-scale AI networks.

A condition where a stalled packet or flow prevents unrelated traffic behind it from progressing, degrading throughput. Virtual output queues and per-priority flow control mitigate it in switches.

A family of technologies enabling data to move directly between GPUs and network or storage devices, bypassing host memory copies to reduce latency and CPU load in AI pipelines.

A high-bandwidth direct interconnect between GPUs and, in newer generations, between GPU and CPU, allowing fast peer-to-peer memory access that outperforms PCIe for multi-GPU communication.

A switch chip that connects multiple GPUs within a server or rack at full NVLink bandwidth, enabling all-to-all communication among GPUs without PCIe bottlenecks.

Peripheral Component Interconnect Express is the standard host expansion bus connecting GPUs, NICs, and storage. Its lane count and generation determine available bandwidth for host-side data movement.

A GPU server board platform integrating multiple high-end GPUs with NVLink and NVSwitch plus high-speed networking, serving as a building block for AI training systems.

An integrated AI system combining GPUs, high-speed interconnects, storage, and optimized software stack, designed to be deployed as a ready-to-run node in training clusters.

A reference architecture for scaling many DGX or HGX nodes into a large cluster with a defined fabric, storage, and management design that meets target performance for large models.

The electrical power consumed per rack, expressed in kilowatts. High-density GPU racks require careful power distribution, cooling design, and facility planning to avoid thermal or electrical limits.

Cooling approach using direct-to-chip cold plates or immersion to remove heat from high-power GPUs more efficiently than air, enabling denser racks and reduced facility energy use.

The maximum heat a component is specified to dissipate under normal operation, used to size cooling systems and plan data center airflow and liquid-loop capacity.

An open RESTful API standard for managing servers, switches, and other data center hardware, enabling out-of-band automation of inventory, health, and firmware operations.

A dedicated service processor on a server that provides out-of-band access for power control, sensor monitoring, and remote console, independent of the host operating system.

Specialized GPU execution units that perform matrix multiply-accumulate operations at high throughput, accelerating deep learning training and inference when using supported precision modes.

A training technique that combines lower-precision arithmetic for most operations with higher-precision accumulation or master weights, improving speed and memory use while preserving model accuracy.

An 8-bit floating-point format used on recent GPUs to accelerate training and inference with reduced memory footprint, typically requiring scaling strategies to maintain numerical stability.

Brain Floating Point 16 is a 16-bit format with wide exponent range and reduced mantissa, widely used for deep learning because it resists overflow while halving memory versus FP32.

A neural network architecture based on self-attention that processes sequences in parallel, forming the basis of modern large language models and many vision and multimodal systems.

A computation that weights each input token by relevance to others, enabling models to capture long-range dependencies. Its quadratic cost drives much AI infrastructure optimization.

Stored key and value tensors from prior tokens that avoid recomputation during autoregressive generation. Its memory footprint grows with context length and batch size, limiting inference throughput.

An attention algorithm that tiles computation in on-chip memory to reduce memory reads and writes, improving speed and enabling longer context lengths without approximate attention.

A collective operation in which every participant ends with the same reduced result, such as summed gradients. It dominates communication time in data-parallel training.

A collective operation where every participant sends distinct data to every other participant. It is central to mixture-of-experts and tensor-parallel workloads and stresses network bisection bandwidth.

NVIDIA Collective Communications Library provides optimized implementations of collective operations across GPUs and nodes, automatically selecting algorithms and topologies for the available interconnect.

A training strategy that replicates the model on each device and splits input batches, synchronizing gradients through all-reduce so replicas stay consistent.

A strategy that splits individual layers' weight matrices across devices, requiring frequent activation exchange within layers and favoring high-bandwidth intra-node interconnects.

A strategy that assigns consecutive model layers to different devices and streams micro-batches through them, trading pipeline bubbles for reduced memory per device.

A strategy for mixture-of-experts models that distributes different expert subnetworks across devices, requiring all-to-all routing of tokens to their selected experts.

An inference serving technique that dynamically adds and removes requests from a running batch at each decoding step, raising GPU utilization versus static batching.

The process of running a trained model to produce predictions or generated output. It emphasizes latency, throughput, and memory efficiency rather than gradient computation.

The iterative process of adjusting model parameters using forward and backward passes over data, requiring large compute, memory bandwidth, and coordinated multi-GPU communication.

An open-source library for optimizing and serving large language models on NVIDIA GPUs, providing optimized kernels, quantization support, and efficient batching for inference.

A set of containerized microservices that package optimized inference engines and model metadata behind standard APIs, simplifying deployment of generative AI models on NVIDIA infrastructure.

A lightweight, isolated runtime package bundling an application with its dependencies. GPU containers rely on the container toolkit to expose device drivers and hardware to workloads.

An orchestration platform for scheduling and managing containerized workloads across clusters. GPU scheduling requires device plugins and resource requests to allocate accelerators correctly.

A Kubernetes operator that automates deployment and lifecycle of GPU software components such as drivers, container toolkit, device plugin, and monitoring on cluster nodes.

Multi-Instance GPU partitions a single GPU into isolated instances with dedicated memory and compute slices, improving utilization for smaller workloads and providing fault isolation.

Centralized monitoring, provisioning, and updating of many servers and accelerators across sites, typically using APIs and telemetry to maintain health and software consistency.

A component distributing incoming requests across multiple backend instances to improve availability and utilization, often used in front of inference service replicas.

A versioned repository storing trained model artifacts, metadata, and lineage, supporting reproducible deployment and governance across environments.