NCCL All-Reduce at 40% of Peak on an InfiniBand Fabric
You operate a 128-node, 1,024-GPU training cluster with NDR 400 Gb/s InfiniBand in a rail-optimized leaf-spine fabric. Each node has eight GPUs and eight HCAs, one per rail. A 512-GPU NCCL all-reduce benchmark reaches only 90 GB/s per GPU versus the 180 GB/s expected from the vendor. ibdiagnet reports symbol errors on three links, and one leaf uplink is running at 200 Gb/s (2x instead of 4x). The subnet manager log shows repeated path migration events. NCCL_IB_HCA is unset, so NCCL discovers all HCAs including a dual-port storage HCA. NCCL_DEBUG=INFO shows the job using mlx5_0 through mlx5_7 plus mlx5_8. The application team says the code is unchanged and worked at full speed last quarter.
1. Is this a fabric problem or an application/configuration problem? What controlled tests would you run first?
Read the worked answer
Treat it as a fabric problem until a controlled test proves otherwise, but do not ignore NCCL configuration. The 200 Gb/s leaf uplink is a hard capacity reduction on one rail, and symbol errors on three links imply retransmissions and effective bandwidth loss. Repeated path migration events suggest the subnet manager is reacting to unstable links, which can cause transient congestion and rank-to-rank variance. However, NCCL discovering mlx5_8, a storage HCA, can also cause incorrect rail selection and extra QPs. Run a ladder of tests: single-node 8-GPU all-reduce to establish GPU and NVLink health, then 8-node, 64-node, and 512-GPU tests with NCCL_IB_HCA explicitly set to the eight compute HCAs. Compare bus bandwidth per GPU and look for variance by rank. If the 8-node test is near peak and the 512-GPU test collapses, the problem is fabric or routing at scale. If even 8 nodes underperform with the storage HCA excluded, check NCCL algorithms and environment. This ladder localizes the fault before any hardware replacement.
2. How do you interpret the symbol errors, link width, and path migration evidence, and what do you prioritize?
Read the worked answer
Symbol errors are physical-layer errors; a few can be normal, but errors on three links alongside a 2x-width uplink are not. A 400 Gb/s NDR link running at 200 Gb/s means only two of four lanes are active, halving that uplink's capacity. If that uplink carries a rail used by the 512-GPU job, all-reduce traffic on that rail becomes the bottleneck and NCCL will report lower bus bandwidth across all ranks. Path migration events are a symptom: the SM is moving traffic away from a failing path, but migration itself can cause brief congestion and uneven routing. Prioritize the physical link first: inspect transceivers, replace cables/optics on the three erroring links and the degraded uplink, and verify link width and BER after replacement. Then check SM configuration and routing policy. Do not simply clear counters; capture a baseline before and after. Also verify that the rail-optimized cabling still matches the intended GPU-to-leaf mapping, because a mis-cabled rail can present as a mysterious bottleneck without any error counters.
3. What remediation and acceptance criteria would you use before returning the fabric to production?
Read the worked answer
Remediate in order: replace or reseat the failing optics and cables, restore the leaf uplink to 4x width, and confirm zero symbol errors over a sustained soak. Then set NCCL_IB_HCA to the eight compute HCAs explicitly, excluding the storage HCA, and confirm NCCL uses the intended rail per GPU. Run NCCL all-reduce at 8, 64, 256, and 512 GPUs with fixed message sizes and record bus bandwidth, rank variance, and completion time. Acceptance should require 512-GPU bandwidth within an agreed percentage of the vendor baseline, no path migration storms during the test, and clean IB counters after the run. Also run a representative training job for several hours, because benchmarks may not reproduce the application's communication pattern. Finally, add monitoring for symbol errors, link width, and SM path migrations, with alerts before degradation reaches the job. The goal is not just to clear the benchmark but to prove the fabric sustains the training roadmap.