I85PREP
⌘ K
300-640 DCAI study materials
300-640 DCAI study resources

300-640 DCAI: High-Speed AI Networking and Fabric Design

300-640 DCAI · Work through your exam, one topic at a time.

All guide sections

High-Speed AI Networking and Fabric Design

This domain tests the lossless Ethernet design that carries RDMA: RoCEv2 classification, PFC, ECN, MTU consistency, topology choice, and the operational telemetry that keeps a congested fabric from collapsing.

Core rules

  • RoCEv2 encapsulates RDMA in UDP destination port 4791. NICs tag it with a DSCP value (commonly 26 or 48); the switch maps DSCP to CoS and to a dedicated QoS group.
  • PFC (802.1Qbb) provides per-priority pause so that a congested egress queue signals the sender instead of dropping. Enable it only on the RoCE class - turning it on everywhere creates head-of-line blocking.
  • ECN plus DCQCN is the congestion-control loop. The switch marks CE bits in the egress queue; the NIC reacts and slows the flow. PFC is the safety net, ECN is the steering wheel.
  • MTU must be consistent end to end, typically 9216. One 1500-byte hop silently drops jumbo frames.
  • Topology: rail-optimized or rail-aligned leaf-spine for training, standard spine-leaf for inference and general workloads. Use per-packet dynamic load balancing where available to spread flows across equal-cost uplinks.

Worked example: lossless policy

The requirement is lossless RoCEv2 for GPU servers on a Nexus 9000 leaf.

  1. Classify the traffic: match DSCP 26, set CoS 3, assign QoS group 3.
  2. In the network QoS policy, set MTU 9216 and enable pause for CoS 3 only, then attach the policy under system QoS.
  3. Enable PFC watchdog with detection and recovery timers so a stuck or faulty NIC cannot pause the fabric indefinitely.
  4. Set ECN marking thresholds in the egress queues facing GPU servers, and confirm DCQCN is enabled on the NIC. Reasoning: PFC alone guarantees zero drops but propagates backpressure; ECN gives the NIC an early signal to reduce rate before pause frames are needed. You need both, plus the watchdog as the failure containment.

Traps that cost marks

  • Enabling PFC without a watchdog, so one bad NIC can stall a whole leaf.
  • ECN thresholds set so high nothing is ever marked, or so low every packet is marked.
  • Mixing 1500 and 9216 MTU across the path.
  • Quoting non-blocking while uplinks are oversubscribed.
  • No telemetry: without per-queue and per-flow counters, congestion is invisible until jobs slow down.