NVIDIA NCP-AII common mistakes
NVIDIA NCP-AII · Understand the traps and how to avoid them.
Treating GPU-to-CPU topology as uniform and sizing PCIe lanes or NUMA pinning from the spec sheet instead of the actual slot-to-CPU mapping, then misjudging which GPUs share a PCIe switch or root complex.
Candidates memorize bandwidth numbers (PCIe Gen4 x16, Gen5 x16) but never practice reading real topology output, so they assume every GPU has equal, dedicated host bandwidth and miss shared-switch or bifurcated-slot constraints.
What to do instead
Practice mapping topology with nvidia-smi topo -m and lspci -tv on real or simulated hosts before answering: identify NV# links, PIX/PXB/PHB/SYS relationships, and which CPU socket each GPU hangs off. Learn to translate PIX vs PHB vs SYS into relative latency/bandwidth expectations and use that to answer placement questions.
Configuring GPU persistence mode, MIG, or ECC state as a one-time step and forgetting that these settings do not survive reboot or driver reload in the same way, then answering questions as if the setting is permanent.
Candidates conflate persistent daemon behavior, persistence mode, and configuration persistence. They assume nvidia-smi -pm 1 or a MIG profile is stored in firmware rather than needing a systemd unit or boot-time script.
What to do instead
Learn which GPU settings are volatile: persistence mode, application clocks, MIG configuration, and compute mode all reset on reboot or driver reload unless reapplied. Study how to persist them via systemd service, nvidia-persistenced, or nvidia-smi mig -cgi at boot, and know which settings ECC changes require a reboot to take effect.
Assuming InfiniBand and Ethernet RDMA behave identically for congestion control and failover, and picking RoCEv2 settings without accounting for PFC, ECN, and lossless configuration requirements.
Candidates learn 'RDMA is RDMA' and apply InfiniBand mental models (credit-based flow control, subnet manager) to Ethernet, missing that RoCEv2 requires lossless fabric configuration, PFC/ECN tuning, and different congestion management.
What to do instead
Separate the two stacks explicitly: InfiniBand uses a subnet manager, credit-based flow control, and per-lane QoS; RoCEv2 requires PFC (or ECN-based congestion control), proper DSCP/priority mapping, and MTU consistency end to end. Practice reading ibstat, iblinkinfo, and perftest output for IB, and ethtool, cma_roce_mode, and PFC counters for RoCE, and know which tool answers which fabric question.
Sizing storage for aggregate bandwidth only and ignoring metadata, small-file, and checkpoint-write patterns, then choosing a parallel filesystem or local NVMe layout that collapses under checkpoint bursts.
Candidates focus on sequential read throughput numbers and overlook that AI workloads are write-heavy at checkpoint time, metadata-intensive during dataset shuffling, and sensitive to client-side caching and striping.
What to do instead
Study workload-specific storage patterns: large sequential reads for training data, burst writes for checkpoints, and metadata-heavy small-file access for inference and preprocessing. Learn how striping, client caching, and parallel filesystem layout (Lustre/GPFS/WEKA-style) affect each pattern, and practice matching a workload profile to a storage tier rather than to a single throughput figure.
Installing the CUDA toolkit and driver in the wrong order or mismatching versions, then assuming any recent driver works with any CUDA toolkit or container image.
Candidates treat the driver, CUDA toolkit, and container CUDA runtime as interchangeable. They miss the compatibility rules: the driver must be new enough for the toolkit, and container images bundle their own runtime but rely on the host driver.
What to do instead
Memorize the compatibility model: driver version sets the maximum CUDA runtime supported; the toolkit and container runtime must be at or below that ceiling; minor-version compatibility and forward compatibility have specific limits. Practice reading nvidia-smi's CUDA Version field, the release notes compatibility matrix, and nvidia-container-toolkit configuration, and know how to diagnose 'CUDA driver version is insufficient' versus a missing library.
Treating container GPU access as automatic and skipping the container runtime configuration, then failing to explain why a container sees no GPUs despite a working host driver.
Candidates assume Docker or an orchestrator passes GPUs through by default. They do not practice configuring the NVIDIA container runtime, the device request syntax, or the CDI-based approach, so they cannot reason about the failure path.
What to do instead
Learn the full chain: nvidia-container-toolkit installed, runtime registered in the container engine config, and the workload requesting GPUs explicitly (device requests or CDI). Practice the diagnostic sequence from host nvidia-smi to container nvidia-smi, including checking the runtime config file and environment variables, and know how GPU visibility differs across container engines and orchestrators.
Assuming a Kubernetes GPU node schedules GPUs without the device plugin and resource advertisement, then misdiagnosing pending pods as a scheduler or image problem.
Candidates know Kubernetes scheduling but not the GPU-specific extension points. They miss that the device plugin advertises nvidia.com/gpu resources, that the GPU operator installs and manages the stack, and that taints, labels, and node selectors gate placement.
What to do instead
Trace the full scheduling path: node labels and taints, device plugin registration, resource advertisement, pod resource requests, and the GPU operator's component stack (driver, container toolkit, device plugin, DCGM exporter). Practice reading pod events and node describe output to distinguish 'no GPU resource advertised' from 'resource exhausted' from 'taint not tolerated'.
Using nvidia-smi as the only monitoring signal and declaring a node healthy when the GPU is visible, ignoring XID errors, ECC errors, thermal throttling, and fabric manager or NVLink faults.
Candidates equate 'nvidia-smi runs' with 'GPU is healthy'. They do not practice reading XID codes, retired pages, throttle reasons, or NVLink error counters, so they miss degraded-but-functional hardware.
What to do instead
Build a layered health check: nvidia-smi for presence and utilization, query-gpu for ECC and retired pages, throttle reasons for clock limiting, nvidia-smi nvlink for link errors, and DCGM or dcgmi diag for deeper validation. Learn common XID meanings and which indicate driver-level versus hardware-level faults, and practice mapping a symptom to the correct diagnostic layer.
Treating security as network perimeter only and leaving GPU and BMC management interfaces, driver signing, and container image provenance unaddressed in cluster designs.
Candidates apply general security instincts but not AI-cluster-specific surfaces: BMC/IPMI exposure, GPU firmware update channels, untrusted container images with host driver access, and secrets in job definitions.
What to do instead
Enumerate the AI-cluster attack surface explicitly: BMC/management network isolation, firmware and driver update integrity, container image signing and registry controls, least-privilege GPU access, and secrets handling in orchestration. Practice evaluating a design for each surface and know which controls are native to the platform versus requiring external tooling.
Validating a cluster with a single-GPU benchmark and declaring it production-ready, skipping multi-node collective tests, fabric validation, and burn-in under sustained load.
Candidates run one nvidia-smi or a trivial CUDA sample and stop. They do not practice the acceptance sequence: per-node burn-in, intra-node NVLink bandwidth, inter-node RDMA bandwidth and latency, and collective operations across the full fabric.
What to do instead
Adopt a staged acceptance plan: per-node GPU burn-in and ECC check, intra-node bandwidth tests, inter-node perftest for RDMA, NCCL all-reduce across the full topology, and sustained-load soak with monitoring. Practice interpreting results against expected fabric bandwidth and knowing which stage isolates a node, a link, or a fabric-wide misconfiguration.