AI Infrastructure
The AI Infrastructure domain checks whether you can design and validate the network and compute foundation AI workloads depend on, using Cisco building blocks: Nexus 9000 and Silicon One-based switching, Nexus Dashboard for fabric management and telemetry, UCS X-Series and C-Series GPU servers managed through Intersight, and Cisco AI Pod validated designs. Most items are design decisions - topology, congestion control, telemetry - rather than raw CLI recall.
Back-end fabric rules
- Training traffic is collective (all-reduce/all-gather) and bursty: the back-end fabric must be non-blocking or close to 1:1 oversubscription. Front-end, storage, and management fabrics can tolerate 3:1 or worse.
- Build EVPN-VXLAN over a routed eBGP underlay. Spanning tree has no place in an AI fabric.
- Lossless Ethernet for RoCEv2 means a no-drop QoS class with PFC, plus ECN marking so senders slow down before pause frames are needed.
- Rail-optimized cabling: GPU NIC index n on every server connects to the same leaf set (its "rail"), keeping collectives one hop away where possible.
- Jumbo MTU end to end, including host NICs, virtual switches, and every transit link.
- Stream telemetry and PFC/ECN counters into Nexus Dashboard Insights; microbursts are invisible to five-minute interface polling.
Worked example
A 64-server cluster has 8 x 400G RoCEv2 NICs per server = 512 host ports. Each leaf offers 64 x 400G downlinks, so eight leaves are needed. If each leaf uplinks only 16 x 400G, oversubscription is 4:1 - acceptable for storage, fatal for training. Reasoning: in all-reduce every GPU transmits and receives at once, so a 4:1 uplink becomes the bottleneck and stretches job completion time. Fix the design first (rail-optimized with 64 uplinks per leaf, or a two-tier spine where uplinks equal downlinks), then configure the lossless class: map the RoCEv2 priority to a no-drop queue, enable PFC on that priority only, and set ECN minimum/maximum thresholds so switches mark packets before they resort to pauses.
Traps that cost marks
- Carrying a leaf-spine habit of 3:1 or 4:1 oversubscription into the back-end fabric.
- Enabling PFC without ECN, producing pause storms, head-of-line blocking across the entire no-drop class, and occasional deadlock.
- Setting MTU on switches but not on host interfaces or the hypervisor.
- Blaming the fabric when a single degraded GPU or NIC is stalling the collective.
- Treating Nexus Dashboard as a configuration console only; the exam expects telemetry and assurance reasoning.