GPU Training Fabric Across Two Halls
A university research group runs a 512-GPU H100 training cluster split between two halls 1.8 km apart. The spine is Cisco Nexus 9364D-GX2A with 400G links; NCCL all-reduce depends on RoCEv2, so the fabric must be lossless. The current design stretches VLAN 100 across both halls over OTV, and jobs show 30% straggler time plus occasional PFC pause storms that freeze every GPU in both halls. The cloud team wants to burst inference to AWS us-east-1 during peak demand, but the only egress is a single 10G Internet circuit with no SLA. The budget can fund either a second 10G Internet circuit or a 10G AWS Direct Connect hosted connection, not both. The CIO asks whether OTV should be replaced, how to contain pause storms, and which cloud path to fund.
1. Should the team keep OTV or migrate to VXLAN EVPN or routed L3? Give a recommendation and the evidence you would collect before committing.
Read the worked answer
Recommend replacing OTV with a routed VXLAN EVPN fabric, or pure L3 between halls, because OTV stretches a single L2 failure and PFC domain. The 1.8 km fiber adds roughly 9 µs one-way, so distance is not the problem; flooding, unknown unicast, and pause propagation are. Evidence to collect: NCCL all-reduce completion time with and without L2 stretch, PFC pause counters per port, ECN marking rates, and the number of hosts in VLAN 100. If an application truly needs L2 adjacency, use VXLAN EVPN with distributed anycast gateways and keep the underlay L3; otherwise route between halls with BGP and place RoCEv2 on routed ports. This reduces blast radius and lets each hall be maintained independently. The decision should be based on measured straggler time, not on the assumption that L2 stretch is required for GPU-to-GPU traffic.
2. Design the PFC/ECN and QoS changes that would contain a pause storm without breaking RoCEv2.
Read the worked answer
Use PFC watchdog, ECN, and separate queues. Configure lossless queues only for RoCEv2 traffic on a dedicated priority, enable ECN marking at thresholds, and tune PFC headroom per port speed. PFC watchdog should detect a paused port and drop or recover it instead of propagating pauses across the fabric. Bound the blast radius by not stretching L2, using per-hall fabrics, and monitoring pause duration and buffer utilization. Set QoS so storage and management traffic cannot consume lossless buffers. Validate with a canary training job and watch for pause storms under incast. The reasoning is that PFC is hop-by-hop and can propagate; ECN avoids pauses before they start, and watchdog limits the damage when a NIC or cable misbehaves.
3. Which cloud connectivity option should be funded for the AWS burst: a second 10G Internet circuit or a 10G Direct Connect hosted connection? Explain the trade-offs.
Read the worked answer
Compare the options against the burst requirement, SLA, and failure modes. A second 10G Internet circuit gives redundancy but no AWS SLA and variable latency; Direct Connect hosted 10G gives predictable latency and an SLA but is a single logical connection unless dual. Cloud OnRamp automates either path but does not create bandwidth. For 512 GPUs, an inference burst can be many Gbps, so 10G is marginal. Recommend Direct Connect 10G with BGP and IPsec or MACsec, use Cloud OnRamp for orchestration, and keep the existing Internet circuit as backup. If the burst is small and cost-sensitive, a second Internet circuit is acceptable, but set expectations on jitter and measure the burst profile first.