Rail-Optimized NDR Fabric Degrades After Node Expansion
A 128-node training cluster (8 GPUs per node, 1,024 GPUs total) runs a 70B-parameter model over an NDR 400 Gb/s InfiniBand fabric in a rail-optimized layout: GPU k on every node connects to leaf k, with eight leaves per spine plane and two spine planes. After 32 new nodes were added using the same reference design, all-reduce step time rose from 41 ms to 49 ms. Fabric telemetry shows no link-down events, but three links on rail 5 report a 12% increase in symbol errors, and the subnet manager master is still the oldest leaf switch, running firmware two releases behind the rest. Some jobs log occasional ibv_poll_cq timeouts. Power and cooling are within budget, and GPU clocks remain at boost. The team is debating whether to roll back the new nodes or tune the fabric.
1. What diagnostic sequence would you run first, and which observations would separate a physical-layer fault from a fabric-control-plane fault?
Read the worked answer
Start at the physical layer because symbol errors are a hardware-facing counter, not a routing symptom. Pull per-port counters for all three suspect links: symbol errors, effective BER, link error recovery, FEC corrected/uncorrected blocks, and whether errors are increasing monotonically or clustered at job start. Compare the new nodes against an older node on the same rail, then swap optics and cables between a suspect and a known-good port. If the errors follow the cable or transceiver, the fault is physical; if they stay with the switch port, escalate to the switch or its firmware.
In parallel, check the control plane. The SM master running old firmware is a credible cause of route programming inconsistency, especially if adaptive routing or SHARP configuration changed with the expansion. Look at SM priority, master election history, LID route distribution, and whether the new nodes were discovered under a different SM configuration. ibv_poll_cq timeouts can come from link recovery or from SM reconfiguration, so correlate their timestamps with port error spikes and SM events. A control-plane fault usually affects paths or groups of nodes; a physical fault stays pinned to specific ports.
2. Assume two of the three links are traced to bad transceivers and one to a dirty connector. What remediation and configuration changes would you make before returning the full cluster to production?
Read the worked answer
Replace the failed transceivers and clean or replace the connector, but do not stop at the hardware fix. First, quarantine the affected rail and verify clean counters with a soak test: run ib_write_bw and ib_write_lat between a new node and an old node on that rail, then run an NCCL all-reduce microbenchmark at the same message sizes as the training job. Second, fix the control-plane asymmetry. Move the SM master to a spine switch or a dedicated management node, upgrade the lagging firmware in a maintenance window, and confirm that all leaves run a consistent firmware and SM configuration. Set SM priority explicitly so the master does not float to an old leaf. Third, validate that adaptive routing and any SHARP settings are applied uniformly; the expansion should not have introduced a mixed configuration. Document the cable and optics vendor/lot for the new nodes, because a batch of marginal transceivers is more likely than three independent failures.
3. How would you prove the fix is effective and safe for a 1,024-GPU all-reduce workload without putting the production job at risk?
Read the worked answer
Use a staged canary. Start with a single-node 8-GPU NCCL test, then a 32-node job that includes the new nodes, then a 64-node job crossing both spine planes. Compare bus bandwidth and step time against the pre-expansion baseline at the same message sizes and collective algorithms. Only after those pass, run a short production-like job with checkpointing disabled or with more frequent checkpoints, and monitor per-rail counters, SM events, and PFC or credit-stall indicators if present.
Define acceptance criteria in advance: no increase in symbol errors over a 24-hour soak, step time within 2% of the 41 ms baseline, no ibv_poll_cq timeouts, and no SM mastership churn. If the full job still shows a gap, compare rail-level throughput to identify a straggler rather than rolling back the entire expansion. Rollback of 32 nodes is a blunt instrument; a rail-level quarantine plus firmware alignment is reversible and preserves capacity.