NVIDIA NCA-AIIO common mistakes
NVIDIA NCA-AIIO · Understand the traps and how to avoid them.
Comparing GPUs by a single peak-FLOPS number from a spec sheet without noting the precision mode or whether the figure is dense or sparse - e.g., treating an FP8 sparse number as if it were the FP16 dense throughput, or comparing an FP64-heavy part to an FP16-heavy part as if they were interchangeable.
Vendor spec pages list several precisions (FP64, TF32, FP16/BF16, FP8, INT8) and often quote sparsity-boosted numbers in the headline. Candidates memorize one number and then answer architecture-fit questions using it, which inverts the ranking for workloads that are FP64-bound versus FP8-bound.
What to do instead
Rebuild your mental table as rows = precision (FP64, TF32, FP16/BF16, FP8/INT8), columns = dense vs sparse, and force yourself to state the precision out loud before choosing a GPU for a scenario. Also learn that TF32 changes the compute datapath only - FP32 values are still stored as FP32.
Applying training rules of thumb to inference questions - arguing that a serving cluster needs NVLink and InfiniBand everywhere, or sizing an inference node by its interconnect bandwidth rather than by memory capacity and memory bandwidth.
Study material for large-model training is dominated by scale-out interconnect discussion, so the training picture gets over-learned and then misapplied. Inference is usually limited by memory bandwidth per token, KV-cache capacity, and latency/batching behaviour, not by inter-GPU fabric throughput.
What to do instead
For any scenario question, first classify it as training (throughput-oriented, gradient sync heavy, needs fast scale-out) or inference (latency/tail-latency-oriented, KV-cache memory bound, batching and quantization sensitive). Choose the hardware and topology from that classification, not from the training default.
Naming a parallelism strategy without matching it to the constraint - recommending tensor parallelism for a model that fits on one GPU, or recommending pipeline parallelism for a model whose layers already fit but whose per-layer weights do not.
Data, tensor, and pipeline parallelism are usually memorized as a list of three terms rather than as three answers to three different bottlenecks (throughput scaling, per-layer weight splitting, depth splitting). The result is a plausible-sounding but wrong answer that ignores the interconnect and bubble trade-offs.
What to do instead
Drill the decision as a chain: does the model fit on one device (use data parallel / replicas)? If not, do individual layers fit (tensor parallel across NVLink-connected GPUs within a node)? If a single layer still does not fit, or the model spans nodes (pipeline parallel, accept pipeline bubbles)? State the bottleneck before naming the strategy.
Sizing accelerator memory from parameter count alone - computing '7B parameters × 2 bytes = 14 GB, so a 16 GB GPU is fine' and ignoring optimizer states, gradients, activations, and inference KV cache.
The parameter-count arithmetic is simple and satisfying, so it becomes the whole calculation. In mixed-precision training with Adam, per-parameter memory is many times the weight size, and in serving, KV cache scales with batch size, sequence length, layers, and head dimension, which often exceeds the weights themselves.
What to do instead
Use a two-bucket budget: static (weights + optimizer states + gradients) plus dynamic (activations for training, KV cache for serving). For serving, compute KV bytes per token per layer from layers × 2 (K and V) × head count × head dim × bytes, multiply by max context × concurrent sequences, and add that to weight memory before comparing to GPU or MIG-slice capacity.
Using NVLink/NVSwitch and InfiniBand/RoCE as synonyms in design answers - describing scale-up (GPU-to-GPU within a node) and scale-out (node-to-node) as the same fabric, or assuming NVLink removes the need for a fast cluster network.
Both are 'fast GPU interconnects' and both appear in DGX-style diagrams, so they collapse into one concept. The consequence is designs that claim all-reduce across hundreds of GPUs will behave like intra-node traffic, or that a storage/RDMA requirement is satisfied by NVLink.
What to do instead
Keep a strict two-column map: scale-up = NVLink/NVSwitch (or PCIe when NVLink is absent) inside a node; scale-out = InfiniBand or RoCE/Ethernet with GPUDirect RDMA between nodes. When a question involves more than one node, the answer must name a network adapter and fabric, not NVLink.
Placing GPUs and their feeding processes arbitrarily across sockets and PCIe switches, and sizing a node's local storage without regard to how fast data must reach the GPUs.
Topology and I/O are invisible on a spec sheet, so candidates treat a multi-GPU server as a homogeneous box. In practice a GPU on a different NUMA node or behind a PCIe switch hop, or a training job reading from a network filesystem with no local NVMe staging, produces stalls that no amount of GPU compute fixes.
What to do instead
In any node-design or troubleshooting scenario, explicitly check GPU-to-CPU and GPU-to-GPU locality with topology inspection tooling, pin processes to the correct NUMA node, and budget local NVMe (plus GPUDirect Storage where supported) for data staging rather than assuming remote storage keeps the GPUs fed.
Choosing multi-tenant GPU sharing by habit - defaulting to time-slicing for production tenants that need memory isolation and predictable latency, or promising a MIG partition on a GPU that does not support MIG.
GPU sharing is often learned as 'time-slicing = sharing', and MIG support is assumed to be universal across data-center GPUs. In reality MIG isolation exists only on selected data-center parts, and time-slicing shares memory with no fault or QoS isolation, which is exactly wrong for a noisy-neighbour or SLA scenario.
What to do instead
Memorize the decision rule: hardware partitioning with memory and fault isolation (MIG, on supported data-center GPUs) for production multi-tenant or SLA-bound work; time-slicing or oversubscription for development, bursty, or best-effort workloads. Before answering, confirm the named GPU is MIG-capable, and state the isolation level the scenario actually requires.
Monitoring GPU health with a single utilization percentage from an interactive query and treating a low or high number as a verdict, without checking memory usage, throttle reasons, ECC errors, or XID events.
Interactive GPU status output is the most familiar tool and its headline utilization figure looks authoritative. But that figure is a sampling of busy-ness, not a measure of useful work, and it will happily read high on a stalled kernel or read moderate on a thermally throttled card.
What to do instead
Build a fixed diagnostic checklist for any 'GPUs are slow / jobs fail' scenario: per-GPU memory versus capacity and any pending-retirement condition, throttle/power/thermal clock reasons, ECC error counters, and XID error codes in the system log. Answer with the metric that discriminates the hypotheses, and describe continuous telemetry collection to a time-series backend rather than ad-hoc queries.
Debugging a container that cannot see the GPU by guessing at the scheduler, when the actual gap is the runtime path - container toolkit not installed or not registered as the runtime, no device request passed to the container, or a stale visible-devices variable shadowing it.
Container-to-GPU plumbing spans several layers (toolkit install, runtime registration, container creation flags, environment variables), and candidates usually only check the outermost one. The failure then gets misattributed to the cluster or the application.
What to do instead
Work the chain in order and name the layer you are testing: is the device present on the host at all; is the container runtime integrated so containers can be granted devices; did the container request the devices (device flag or an equivalent resource request in the orchestrator); is the visible-devices variable restricting access; does the in-container driver library version match the host driver? Answer the scenario at the layer that is actually broken.
Assuming version compatibility in the wrong direction - running an application built against a newer CUDA major version and expecting it to work on an older installed driver, or letting a driver-management component fight a driver that was already installed on the node.
'Newer app on older driver' feels safe because forward compatibility is a familiar phrase, and driver-management tooling is assumed to be idempotent. Both assumptions produce answers that break at runtime: the driver ships with a maximum supported CUDA version, and installing a managed driver on top of a pre-existing one creates conflicting installations.
What to do instead
Before answering, pin three versions and check them against each other: application build version, installed driver's supported maximum version, and the managed/physical driver's supported versions. State that the driver must be new enough for the built version, and when a node already has a driver, describe explicitly how you avoid installing a second one rather than assuming the tooling sorts it out.