Networking for AI Clusters (InfiniBand and Ethernet)
What this domain tests
Whether you can design, verify, and debug the fabric that carries collective communication. Expect rail-optimized topology, subnet manager operation, RoCEv2 lossless configuration, NCCL transport selection, and link-level counters.
Core rules
- Rail-optimized design. In an eight-GPU node, each GPU pairs with its own leaf switch rail so that GPU n on every node shares one leaf. This keeps all-reduce traffic on a single hop and is the reason physical cabling must match the NCCL rail assumption.
- Subnet manager. InfiniBand needs exactly one active SM per fabric - OpenSM or UFM. Two SMs with equal priority cause mastership fights and intermittent connectivity.
- RoCEv2 needs lossless tuning. Configure PFC on the priority used by RoCE, ECN marking, and a matching DSCP value end to end; mismatched PFC settings are the usual cause of silent drops.
- NCCL transport.
NCCL_DEBUG=INFOprints the chosen transport. SeeingNET/Socketinstead ofNET/IBmeans NCCL fell back, usually from HCA filtering or GID index problems.
Worked example
An eight-node job shows all-reduce bandwidth at roughly one-twentieth of line rate. NCCL_DEBUG=INFO reports socket transport. Cause: NCCL_IB_HCA was unset and the container lacked /dev/infiniband device access, so no verbs device was visible. After exposing the devices and pinning NCCL_IB_HCA=mlx5_0,mlx5_1,..., NCCL reports NET/IB and bandwidth recovers. The reasoning: NCCL silently degrades rather than failing, so transport strings, not error messages, are the diagnostic signal.
Traps that cost marks
Assuming a link is healthy because ibstat says Active - check perfquery symbol-error and link-downed counters too. Forgetting that a misaligned rail cabling turns a one-hop collective into a spine traversal. Setting an MTU larger than the switch supports on RoCE, which fragments and destroys latency.