300-640 DCAI common mistakes
300-640 DCAI · Understand the traps and how to avoid them.
Sizing GPU fabric oversubscription using the ratio of uplinks to downlinks alone, without accounting for the traffic pattern of the collective operations the cluster runs.
The oversubscription ratio is the most quotable number in AI fabric design, so candidates memorize '1:1 for training, 3:1 is fine for inference' and apply it mechanically. They forget that all-reduce and all-to-all generate different fan-in patterns, and that a rail-optimized leaf-spine with an apparently high ratio can still deliver full bisection for the dominant flow.
What to do instead
For every fabric sizing question, first classify the workload (all-reduce-heavy training, all-to-all MoE, parameter-server, or inference with bursty east-west plus north-south storage traffic), then reason about the bisection bandwidth available to the collective's communication pattern. Practice computing per-GPU east-west bandwidth and comparing it to NIC and GPU link capacity before quoting a ratio.
Treating GPU memory capacity as the only sizing input and ignoring memory bandwidth, interconnect (NVLink/PCIe) bandwidth, and host-to-device transfer time when choosing a server or node profile.
Model-size arithmetic (parameters times bytes per parameter) is easy and satisfying, so candidates stop there. Real training throughput is frequently bound by interconnect or host bandwidth, and inference is bound by memory bandwidth for decode, not capacity.
What to do instead
For each design scenario, build a small roofline-style table: FLOPs needed, memory capacity needed, memory bandwidth needed, intra-node interconnect bandwidth, and per-GPU network bandwidth. Justify the node SKU against the binding constraint, and state explicitly which resource is the bottleneck.
Designing the storage tier around capacity and IOPS only, omitting the metadata and small-file behavior of dataset loading, checkpoint writes, and multi-reader shuffling.
Traditional storage design habits center on capacity and random IOPS. AI pipelines stress metadata operations (millions of small files), sequential read of large shards, and periodic synchronized checkpoint bursts from every rank at once.
What to do instead
For each storage question, enumerate the four access patterns (dataset ingest, sequential training reads, checkpoint write bursts, model artifact distribution) and size metadata throughput, read bandwidth, and write burst absorption separately. Call out parallel filesystem or object-store front ends, client-side caching, and checkpoint staggering as mitigations.
Assuming congestion control and QoS tuned for general enterprise traffic will behave the same on RDMA/RoCE AI fabrics, and failing to address PFC storms, ECN marking, and buffer sizing explicitly.
Candidates carry over TCP wisdom (drop-based congestion signals, large buffers) into a lossless or near-lossless fabric where pause frames can propagate head-of-line blocking across the whole spine.
What to do instead
For any RoCE design, state the lossless configuration explicitly: PFC priorities and watchdog, ECN thresholds and marking, DCQCN or equivalent parameters, and buffer allocation per port. Explain how a slow receiver is isolated and how PFC deadlock is bounded.
Presenting automation as 'we will use Ansible and Terraform' without mapping which lifecycle stage each tool owns, how state and inventory are sourced, and how GPU node configuration drift is detected.
Tool names are easy to list and sound complete. Exam scenarios probe the seams: who owns fabric underlay versus cluster provisioning, how node imaging handles driver and CUDA versions, and how idempotency is verified after a partial rollout.
What to do instead
Draw the lifecycle as stages (hardware validation, fabric bring-up, node imaging, cluster bootstrap, workload scheduling, day-2 config) and assign one owner tool per stage with its source of truth. Include drift detection and a rollback path for every automated change.
Answering multi-tenancy questions by isolating only at the Kubernetes namespace level while leaving GPU memory, fabric, and shared filesystem paths unpartitioned.
Namespace isolation is the default mental model from general Kubernetes experience. AI clusters add side channels: MIG or GPU partitioning, NVLink domains, RDMA partition keys, and shared high-performance storage namespaces.
What to do instead
For each tenancy scenario, state the isolation mechanism at every layer: GPU partitioning mode, node pools and taints, network segmentation and partition keys, storage namespace and quota, and identity/RBAC boundaries. Note which layers rely on policy versus hard enforcement.
Designing resiliency around single-component redundancy (dual PSUs, redundant leafs) without analyzing correlated failures such as a whole rack losing power, a top-of-rack switch failure taking an entire rail, or a synchronized checkpoint storm after recovery.
Redundancy checklists are familiar and quick to produce. Correlated and cascading failures require reasoning about blast radius and recovery dynamics, which takes longer under exam time pressure.
What to do instead
For each resiliency question, define the failure domain explicitly (GPU, node, rack, rail, fabric plane, storage target), state the blast radius, the detection mechanism, and the recovery time. Include the thundering-herd behavior on restart and how training job checkpoint/resume limits lost work.
Ignoring power and cooling constraints when specifying rack density, treating the design as a pure network-and-compute exercise.
Candidates focus on bandwidth and GPU count because those are the headline metrics, and forget that high-density GPU racks often exceed conventional per-rack power budgets and require liquid cooling or rear-door heat exchangers.
What to do instead
Always compute per-rack power from node TDP plus switch and optics draw, compare against the facility's per-rack and per-row limits, and state the cooling approach (air, direct-to-chip liquid, immersion) and any derating for altitude or inlet temperature. Mention power capping and utilization-based sustainability metrics where relevant.
Describing orchestration as 'use the GPU operator and scheduler' without explaining how GPUs are advertised, how topology-aware placement works, and how gang scheduling prevents partial job starts.
The components are familiar names, so answers stay at the component level. Exam scenarios test the mechanics: device plugin resource advertisement, topology manager policy, MIG resource naming, and why a distributed job needs all-or-nothing scheduling.
What to do instead
For scheduling questions, trace the path from device plugin advertisement to scheduler decision, and name the policy choices (topology manager scope and policy, NUMA alignment, gang or coscheduling plugin, queue fairness). Explain what happens when a job cannot be placed atomically.
Treating security as a perimeter concern and leaving in-cluster AI assets unaddressed: unencrypted model weights at rest, shared service accounts with broad GPU access, unrestricted egress from training pods, and unvetted model artifacts pulled at runtime.
AI platforms inherit enterprise security templates that were built for web workloads. Model artifacts, notebook environments, and high-bandwidth data paths introduce exposures that those templates do not cover.
What to do instead
For each security scenario, cover identity and least privilege for GPU and data access, encryption in transit for RDMA and storage, artifact provenance and signing, network egress policy for training jobs, and audit logging of model and dataset access. Tie each control to the specific asset it protects.