NCCL Throughput Collapses on a Rail-Optimized InfiniBand Fabric
You operate a 32-node cluster, each node with 8 H100 GPUs and 8 NDR InfiniBand compute NICs in a rail-optimized fat tree: GPU i uses mlx5_i on rail i. Two additional 400G NICs per node serve storage. After adding eight nodes, a 32-node NCCL all-reduce benchmark falls from about 180 GB/s to 45 GB/s. ibstat shows all rail links at 400 Gb/s, but perfquery on leaf-07 port 12 shows rising symbol errors and link_downed; the subnet manager log records repeated link-down events on that port. NCCL_DEBUG=INFO shows the job using mlx5_3 and mlx5_7, not all eight HCAs. No GPU XID errors are present. You must determine whether the fault is fabric, host configuration, or topology mapping, and restore full bandwidth.
1. How do you isolate whether the 45 GB/s result is caused by the fabric, the host NCCL configuration, or incorrect rail mapping?
Read the worked answer
Separate the layers with targeted tests. Run per-rail ib_write_bw between corresponding HCAs on two nodes, including rail 7, to see whether one rail is slow even though ibstat reports 400 Gb/s. If only rail 7 is degraded, the problem is fabric or physical. Next inspect the NCCL environment: if NCCL_IB_HCA is unset or restricted, NCCL may use only a subset of HCAs; the log showing mlx5_3 and mlx5_7 supports that. Set NCCL_IB_HCA=mlx5_0,mlx5_1,...,mlx5_7 and rerun. Finally verify rail mapping with nvidia-smi topo -m and the GPU-to-NIC affinity; a GPU using a NIC on the wrong rail creates cross-rail traffic. A slow rail plus a two-HCA NCCL job can explain the collapse.
2. Interpret the perfquery counters and subnet-manager events. What failure mode do they indicate, and what do you check physically?
Read the worked answer
Rising symbol errors plus repeated link_downed on one leaf port indicate a physical-layer problem, not congestion alone. Symbol errors point to signal integrity: a dirty or damaged cable, a failing transceiver, a bad port, or incorrect FEC negotiation. Link flaps cause the subnet manager to reroute, and adaptive routing may not fully compensate; NCCL collectives then see reduced effective bandwidth or timeouts. Check the cable and transceiver on leaf-07 port 12, clean and reseat the fiber, swap the cable and then the transceiver with known-good parts, and confirm the port is configured for the expected speed and FEC mode. Watch perfquery until symbol errors stop increasing and link_downed is stable.
3. What remediation and validation steps would you take before returning the cluster to production?
Read the worked answer
Replace or repair the suspect physical component, clear counters, and confirm the port stays up with no new symbol errors under load. Verify the subnet manager has one master and that all links are at the expected width and speed. Correct NCCL configuration so all eight compute HCAs are used, and validate rail mapping: GPU i should use mlx5_i. Run per-rail ib_write_bw at 400G, then a 32-node NCCL all-reduce and confirm bandwidth returns near the 180 GB/s baseline. Run ibdiagnet for fabric-wide errors and check GPU XID and DCGM logs. Document the faulty port and cable so the same leaf is not reused without testing.