300-640 DCAI flashcards

32 cards. Read the term and the definition here. The deck at the top of this page is optional.

Fat-tree bisection bandwidth

Total bandwidth crossing the midpoint of a Clos/fat-tree fabric. A 1:1 non-blocking design makes bisection equal to half the aggregate edge bandwidth, so any collective pattern runs without oversubscription.

Oversubscription ratio

Ratio of total downlink capacity at a tier to its uplink capacity. A 3:1 ratio means edge devices can demand three times the uplink bandwidth, which is acceptable for inference but risky for training.

East-west traffic

Traffic between servers inside the AI cluster, dominated by collective communication during training. It far exceeds north-south client traffic, so spine and leaf capacity are sized from it.

Rack power density

Electrical load per rack, computed by summing server, switch, and optic draw plus cooling overhead. GPU racks commonly exceed 50 kW, forcing liquid cooling or rear-door heat exchangers.

Management plane separation

A dedicated network for BMC, switch management, and orchestration traffic, isolated from the GPU data fabric. It prevents control traffic from competing with collectives and limits blast radius.

GPUDirect RDMA

A capability letting a network adapter read and write GPU memory directly over PCIe, bypassing host buffers. It lowers latency and CPU load for multi-node collectives but needs peer-to-peer support.

NVLink domain

The set of GPUs connected through NVSwitch inside one node, giving far higher bandwidth than PCIe. Intra-node collectives stay inside this domain; only inter-node traffic uses the NIC.

MIG partitioning

Multi-Instance GPU splits one physical GPU into isolated slices with dedicated memory and compute. It suits multi-tenant inference but caps the memory available to any single job.

Ring all-reduce cost

Each rank sends and receives roughly twice the message size regardless of GPU count, so bandwidth cost is nearly constant while latency grows with ring depth.

Gradient accumulation

Running several micro-batches before one optimizer step. Effective batch size equals micro-batch times data-parallel ranks times accumulation steps, trading time for memory without changing total compute.

BF16 versus FP16

BF16 keeps FP32's exponent range with fewer mantissa bits, so training rarely needs loss scaling. FP16 has more precision but a narrow range, requiring dynamic loss scaling.

ZeRO sharding stages

A memory-reduction method that partitions optimizer state, then gradients, then parameters across data-parallel ranks. Each stage cuts per-GPU memory further at the cost of more communication.

Parallel file system

A storage system spreading data and metadata across many servers to deliver aggregate bandwidth. Lustre, Storage Scale, and WEKA are typical choices for training datasets and checkpoint bursts.

Checkpoint burst sizing

Checkpoint size equals parameter count times bytes per element, plus optimizer state. Write bandwidth must absorb the full size within the job's tolerable pause, often terabytes in seconds.

Storage tiering

Placing data by access pattern: NVMe parallel file system for active datasets and checkpoints, object storage for curated datasets and model registry, archive for raw ingestion.

Small-file penalty

Metadata operations dominate when datasets contain millions of tiny files, stalling training input pipelines. Packing records into large shards reduces metadata load and improves read throughput.

Dataset snapshot lineage

Recording the immutable dataset version and model registry entry for every training run. It makes results reproducible and supports audit of which data produced which model.

RoCEv2 losslessness

RDMA over Converged Ethernet requires priority flow control and ECN marking with congestion control such as DCQCN. Without them, packet loss causes severe throughput collapse in collectives.

Rail-optimized fabric

A wiring scheme where the same GPU index in every server connects to the same leaf switch. Collectives between matching indices then stay one hop, reducing latency and congestion.

Adaptive routing

Per-packet or flowlet path selection that spreads large flows across many links. It reduces hash collisions that static ECMP suffers when a few elephant flows dominate.

InfiniBand NDR speed

The 400 Gb/s per-port InfiniBand generation, following HDR at 200 Gb/s. Port-count and optics math must state the generation because Ethernet has parallel 400G products.

PFC pause frames

Priority Flow Control messages that temporarily stop a traffic class to avoid buffer drops. Rising pause counts are an early congestion signal on a lossless AI fabric.

Optics reach classes

Transceiver grades defined by distance: DR4 around 500 m single-mode, FR4 around 2 km, and DAC or AEC for in-rack and short-row links at lower cost and power.

Slurm versus Kubernetes

Slurm schedules batch HPC and training jobs on bare metal with strong topology awareness. Kubernetes with GPU device plugins dominates containerized inference and shared services.

GPU device plugin

A Kubernetes component that advertises GPU resources to the scheduler and configures containers. Jobs request whole GPUs or MIG slices as extended resources with node affinity rules.

DCGM exporter

A Prometheus exporter publishing GPU metrics such as utilization, memory, temperature, ECC errors, and XID codes. It enables alerting on degraded hardware before jobs fail.

Node drain and cordon

Scheduler operations that stop new work and evict running pods or jobs before maintenance. Combined with a canary job, they make upgrades safe on a shared AI cluster.

Infrastructure as code

Declaratively describing fabric, compute, and storage in version-controlled templates. Rebuilds become reproducible and drift between intended and actual state is detectable.

Multi-tenant GPU isolation

Separating tenants by namespace or partition, network policy, and MIG slices. Sharing a GPU across trust boundaries without these controls risks data leakage and noisy-neighbor effects.

Model supply chain risk

Serialized model files can execute code on load. Pinning images by digest, scanning for vulnerabilities, and verifying external weights reduce the risk of compromised artifacts.

Checkpoint restart resiliency

Large jobs fail more often as GPU count rises, so frequent checkpoints plus automatic restart from the last good state bound the lost work to one checkpoint interval.

PUE and GPU efficiency

Power Usage Effectiveness is total facility power divided by IT power. Combined with tokens or samples per kilowatt-hour, it measures both cooling overhead and useful AI work per watt.