NVIDIA NCP-AIO common mistakes
NVIDIA NCP-AIO · Understand the traps and how to avoid them.
Treating power and cooling budgets as an afterthought, sizing racks by GPU count rather than by per-rack power density (for example assuming a standard 10-15 kW rack can host a fully populated HGX/DGX node).
Candidates study GPU specs in isolation and carry over intuition from general data center work where rack power was rarely the binding constraint; AI racks routinely exceed 40-100 kW and require liquid cooling or rear-door heat exchangers.
What to do instead
For each deployment scenario, compute rack power from GPU TDP plus CPU, NIC, and PSU overhead, then check it against the facility's cooling type (air vs. direct-to-chip liquid) and PDU capacity before answering. Memorize the rough power class of current high-end GPU platforms and the cooling method each implies.
Confusing the roles of the different management and orchestration layers, attributing GPU health telemetry to the cluster scheduler or expecting the provisioning tool to handle workload scheduling.
NVIDIA's AI factory stack has overlapping-sounding components (base command manager for provisioning, GPU operator for driver/device plugin lifecycle, DCGM for telemetry, the scheduler for placement), and candidates map them from memory of generic Kubernetes or HPC experience.
What to do instead
Build a one-page layer map: hardware/firmware, OS and drivers, container runtime and GPU Operator, telemetry (DCGM/Fleet Command style tooling), scheduler, and workload. For each practice question, first identify which layer the scenario describes, then answer within that layer.
Assuming GPUs are automatically shared or that any pod requesting a GPU gets a full exclusive device, without reasoning about MIG profiles, time-slicing, or MPS configuration.
Candidates know GPUs are 'allocated' in Kubernetes but do not internalize that the default device plugin grants whole GPUs; fractional sharing requires explicit MIG or time-slicing setup and has isolation and memory trade-offs.
What to do instead
Learn the three sharing modes and their guarantees: MIG gives hardware partitioning with memory and fault isolation, time-slicing gives best-effort concurrency with no memory isolation, MPS gives spatial sharing within a process group. Practice mapping workload types (small inference, large training, multi-tenant dev) to the right mode.
Diagnosing a GPU falling off the bus or an XID error as an application bug and restarting the workload instead of reading the XID code and checking fabric, power, or thermal causes.
XID errors surface as workload crashes, so the visible symptom points at the application; candidates have not memorized which XID classes indicate hardware, driver, or fabric faults and skip the telemetry step.
What to do instead
Memorize the major XID categories: correctable vs. uncorrectable memory errors, fallen-off-the-bus, and NVLink/fabric-related faults. For any GPU-failure scenario, state the triage order: check DCGM health and XID, check NVLink/fabric status, check power and thermals, then escalate to RMA only after ruling out fabric and firmware.
Treating InfiniBand and Ethernet as interchangeable for the AI fabric, or forgetting that RoCE requires lossless configuration (PFC, ECN) to perform like InfiniBand.
Both are 'networking' and candidates apply general Ethernet knowledge; they miss that collective communication traffic is bursty and extremely sensitive to packet loss and congestion, and that RoCEv2 needs explicit lossless tuning.
What to do instead
Contrast the two: InfiniBand with adaptive routing and credit-based flow control vs. Ethernet/RoCEv2 needing PFC and ECN plus careful buffer tuning. For any fabric question, identify whether the workload is collective (all-reduce) or point-to-point, then reason about congestion control and tail latency.
Ignoring topology when placing a multi-GPU job, letting the scheduler spread ranks across nodes or PCIe switches so that NVLink-connected GPUs are not used for the heaviest communication.
Candidates focus on GPU count and memory and assume the scheduler handles locality; they do not reason about NVLink domains, NUMA affinity, or rail-optimized fabric layout.
What to do instead
Practice reasoning about the hierarchy: intra-GPU, NVLink/NVSwitch within a node, PCIe/NUMA within a node, and east-west fabric across nodes. For placement questions, prefer keeping collective-heavy ranks within an NVLink domain and aligning NIC selection with rail-optimized topology.
Assuming a training job's throughput bottleneck is the GPU model, without checking data pipeline stalls, CPU oversubscription, or storage read bandwidth.
Marketing emphasizes GPU FLOPS, so candidates jump to compute; in practice input pipelines, dataloader workers, and storage throughput often starve the GPU, and candidates lack a habit of measuring utilization before optimizing.
What to do instead
Adopt a fixed diagnostic order: measure GPU utilization and SM occupancy, check whether the GPU is idle waiting on data, then check dataloader worker count, CPU affinity, pinned memory, and storage IOPS. For any 'training is slow' scenario, state which metric distinguishes compute-bound from input-bound.
Confusing precision formats and their effects, treating FP8, FP16/BF16, and TF32 as interchangeable or assuming mixed precision always improves throughput without accuracy risk.
The formats look similar and candidates learn them as 'lower precision is faster'; they miss the role of accumulation precision, loss scaling, and which formats specific tensor cores support.
What to do instead
Build a table of formats with mantissa/exponent trade-offs, typical use (BF16 for training stability, FP8 for inference and newer training recipes, TF32 for transparent matmul acceleration), and the accuracy mitigations each needs. Tie throughput claims to tensor core support and memory bandwidth, not just bit width.
Answering capacity or scaling questions by multiplying node count linearly, ignoring scaling efficiency, communication overhead, and memory-per-GPU limits on batch size.
Linear scaling is the intuitive mental model, and candidates have not internalized that all-reduce cost grows with cluster size and that global batch size is constrained by per-GPU memory.
What to do instead
For scaling scenarios, reason explicitly about communication-to-computation ratio, whether the model fits with the chosen parallelism (data, tensor, pipeline), and the resulting efficiency curve. State assumptions about interconnect bandwidth and batch size before giving a number.
Skipping the operational lifecycle around a GPU node: firmware and driver version compatibility, drain and maintenance procedures, and validation after an upgrade.
Candidates study deployment as a one-time install and neglect day-two operations; compatibility matrices and drain/validate workflows are less memorable than architecture facts.
What to do instead
Learn the upgrade sequence as a checklist: verify the driver/CUDA/firmware compatibility matrix, cordon and drain the node, apply firmware and driver in the supported order, reboot, run health validation (DCGM diagnostics, fabric checks), then uncordon. Practice questions that ask 'what is the first step' with this ordering in mind.