NVIDIA NCP-AIN common mistakes
NVIDIA NCP-AIN · Understand the traps and how to avoid them.
Treating InfiniBand and Ethernet as interchangeable when answering questions about lossless fabric, congestion control, and routing, and applying Ethernet assumptions (e.g., standard TCP retransmission behaviour, spanning-tree-style loop prevention) to an InfiniBand design scenario.
Most candidates come from an Ethernet background, so they default to familiar models. InfiniBand uses credit-based flow control, adaptive routing, and a subnet manager, so answers copied from Ethernet logic are subtly wrong, and the question stems often do not flag the technology shift explicitly.
What to do instead
Before answering, explicitly classify the fabric in the stem: InfiniBand (subnet manager, credit-based flow control, adaptive routing, SHARP) versus Ethernet (RoCEv2, PFC, ECN, DCQCN). Build a two-column comparison sheet and rehearse which congestion-control and routing mechanisms belong to each. For every practice question, write the fabric type next to your answer and verify the mechanism you chose exists in that fabric.
Confusing RoCEv1 and RoCEv2 encapsulation details and their consequences, for example assuming RoCEv1 can be routed across Layer 3 or that RoCEv2 requires the same lossless Layer 2 treatment as RoCEv1.
The two versions differ mainly in encapsulation (Ethernet-only versus UDP/IP), and questions test the operational consequence rather than the acronym. Candidates memorise 'RoCEv2 is routable' without connecting it to PFC/ECN configuration, VLAN behaviour, and MTU implications.
What to do instead
Memorise the encapsulation chain: RoCEv1 is EtherType 0x8915 with no IP header and is not routable; RoCEv2 is UDP destination port 4791 over IP and is routable. Then practise explaining what that means for PFC scope, ECN marking, and MTU. Answer questions by first stating which header exists, then deriving the operational requirement.
Designing or evaluating a data centre fabric without accounting for oversubscription ratio, east-west traffic, and rail-optimised topology, instead choosing leaf-spine diagrams purely on port count.
Candidates focus on connectivity (does every node reach every node) and ignore bandwidth convergence. AI training workloads are dominated by all-reduce traffic that is highly sensitive to oversubscription and to rail alignment, so a topology that looks correct on a port-count basis fails on performance.
What to do instead
For any topology question, compute the oversubscription ratio explicitly (downlink bandwidth divided by uplink bandwidth) and state it. Learn the rail-optimised pattern: each GPU server connects its Nth GPU to the Nth leaf, so same-index GPUs across servers share a rail and all-reduce traffic stays on one hop. Practise sketching a two-tier rail-optimised fabric and labelling the ratio at each tier.
Ignoring power, cooling, and physical density constraints when sizing a GPU cluster, such as specifying racks of high-TDP GPUs without checking per-rack power budget or cooling mode.
Candidates treat data centre questions as pure networking or pure compute questions. Modern GPU nodes can draw far more per rack than traditional CPU racks, and the constraint that actually blocks a design is often power delivery or cooling capacity, not switch ports.
What to do instead
Build a simple per-rack budget: sum node TDP plus switch and PDU overhead, compare against the rack power limit, and state whether air or liquid cooling is required. Memorise the rough order of magnitude for high-end GPU nodes (multiple kilowatts per node, tens of kilowatts per dense rack) and always sanity-check a proposed design against those numbers before selecting an answer.
Conflating model training and inference requirements, for example recommending the same GPU configuration, precision, and interconnect for a batch training job and a latency-sensitive inference service.
Both are 'AI workloads', so candidates reuse one mental model. In reality training is throughput- and interconnect-bound with large memory needs, while inference is latency-, batch-size-, and memory-bandwidth-bound, and the optimisation levers differ (for example quantization and batching matter far more at inference).
What to do instead
For each AI workload question, first label it training or inference, then list the binding constraint: training binds on interconnect bandwidth, memory capacity, and sustained FLOPS; inference binds on latency, memory bandwidth, and batch scheduling. Practise mapping techniques such as mixed precision, quantization, tensor parallelism, and continuous batching to the correct phase.
Treating GPU memory capacity and GPU memory bandwidth as the same constraint, and choosing a larger-memory GPU when the workload is actually limited by bandwidth or by interconnect.
Specification tables list both numbers and candidates remember 'more memory is better'. Questions that describe a model that fits but runs slowly are testing bandwidth or communication awareness, and a capacity-only answer is wrong.
What to do instead
Separate the three limits: capacity (does the model plus activations fit), bandwidth (how fast can memory be read per step), and interconnect (how fast can GPUs exchange gradients). For any scenario, ask which limit the described symptom maps to, and justify the choice with the symptom wording, such as out-of-memory versus low utilisation versus long communication stalls.
Assuming a cloud GPU instance delivers the same interconnect performance as an on-premises cluster, ignoring instance-level networking options such as high-bandwidth GPU-to-GPU fabrics and placement constraints.
Candidates compare cloud offerings on GPU model and count only. Providers expose different networking tiers per instance family, and multi-node training performance depends heavily on whether the instances are placed to get the high-bandwidth fabric.
What to do instead
When evaluating a cloud design, enumerate the networking tier explicitly: what GPU-to-GPU bandwidth the instance family offers, whether a specialised fabric or cluster placement is required, and whether the provider supports the same collective operations. Practise writing a short comparison of two instance families covering GPU count, interconnect, and storage throughput, then state which workload each suits.
Confusing orchestration and scheduling layers with the underlying infrastructure, for example attributing GPU scheduling, multi-tenancy, or resource quotas to the wrong component in a Kubernetes-based AI stack.
The AI cloud stack has several overlapping layers (container runtime, orchestrator, GPU device plugin, operator, scheduler) and candidates learn the names without a clear responsibility map, so they pick a plausible but incorrect layer.
What to do instead
Draw a layered diagram and assign one responsibility per layer: hardware and drivers, GPU device plugin exposing resources, scheduler placing pods, operator managing lifecycle, and the platform layer enforcing quotas. For each practice question, name the layer before choosing the answer, and verify the component you selected is the one that actually performs that action.
Answering storage and data-pipeline questions by focusing only on capacity, ignoring throughput and input-pipeline stalls that leave GPUs idle during training.
Candidates think of storage as a sizing problem. In real training clusters, the data loader can starve the GPU, and the fix is throughput, caching, or pipeline parallelism rather than more capacity, so capacity-only answers miss the point of the scenario.
What to do instead
Always compute the required read throughput: samples per second times bytes per sample, and compare with the storage tier's delivered bandwidth. Learn the common remedies (local NVMe caching, prefetching, sharded formats, parallel data loading) and map each symptom, such as low GPU utilisation with high I/O wait, to the right remedy.
Memorising product names and marketing tiers instead of the underlying capability, so when a question describes a capability the candidate cannot map it to a product family or to the correct generation.
Vendor portfolios change frequently and share similar names across compute, networking, and software lines. Candidates who learn only names cannot transfer knowledge to a scenario phrased around a capability, and they confuse adjacent products in the same family.
What to do instead
Build a capability-first index: for each product family, record what problem it solves, which fabric or workload it targets, and one distinguishing feature. Review it by covering the names and recalling them from the capability description. When a question names a product, restate its role in one sentence before evaluating the options.