AI Networking and Interconnect
AI Networking and Interconnect
This domain covers the fabrics that feed GPUs: NVLink/NVSwitch inside a node, InfiniBand or RoCE between nodes, rail-optimized topologies, GPUDirect RDMA, and congestion control.
Core rules and concepts
- Keep GPU east-west traffic on the high-speed fabric; use management Ethernet only for control.
- InfiniBand uses a subnet manager and credit-based flow control; RoCEv2 needs PFC/ECN tuning and lossless design.
- Rail-optimized designs map each GPU/NIC pair to its own leaf rail, maximizing bisection for all-reduce.
- GPUDirect RDMA lets NICs move data directly to GPU memory; it requires IOMMU and driver support.
- Use NCCL tests and per-rail counters to find a bad cable or mismapped NIC.
Worked example
A 64-GPU cluster has eight nodes, eight GPUs per node, and eight rail leaf switches. Each node connects one NIC per rail, so each leaf sees eight downlinks, one per node. If NCCL bandwidth drops after adding a node, compare per-rail counters: one rail at half throughput points to a bad transceiver or a NIC mapped to the wrong rail. The reasoning is that rail symmetry is what makes collective traffic balanced; one asymmetric link becomes the bottleneck.
Traps that cost marks
Putting storage and GPU traffic on the same lossy VLAN; enabling PFC without end-to-end validation; assuming 400G InfiniBand and 400G Ethernet are interchangeable; and testing with TCP instead of RDMA.