Scaling stall on an 8-node Ascend 910B training cluster
An 8-node training job runs on Atlas 800T A2 servers, each with 8 Ascend 910B NPUs and two 200G RoCE NICs. The model is a 7B-parameter dense transformer trained in BF16 with data parallelism only. At 4 nodes (32 ranks), mean step time is 1.90 s and HCCL all-reduce reaches 178 GB/s. At 8 nodes (64 ranks), step time rises to 4.15 s and all-reduce falls to 66 GB/s; AI Core utilization drops from 82% to 47%. Loss still decreases. The ranktable was generated manually after a NIC replacement. Each server has one NIC port on a different leaf switch. MTU is 1500 on three nodes and 4200 elsewhere; PFC is enabled only on the 4200 nodes. The job uses default HCCL environment variables. No NPU ECC errors appear in npu-smi. The team wants to know whether to add more nodes or fix the fabric first.
1. Is the 4-to-8-node slowdown mainly a compute problem or a communication/topology problem? Give the quantitative reasoning and an ordered diagnostic plan that would confirm or rule out the fabric causes before any nodes are added.
Read the worked answer
Compute is not the first suspect. Weak-scaling efficiency from 4 to 8 nodes is 1.90/4.15 = 46%; if global batch was held fixed and per-rank batch halved, strong-scaling efficiency is only about 23%. All-reduce bandwidth fell to 66/178 = 37% of the 4-node value. For 7B BF16 parameters, gradients are about 14 GB; a 64-rank ring all-reduce moves roughly 2*(N-1)/N*14 GB = 27.6 GB per rank, which at 66 GB/s is about 0.42 s. That does not explain the 2.25 s step-time increase, so look for congestion or a straggler as well as collective inefficiency.
Ordered plan: (1) run npu-smi info and HCCN_Tool link/error checks on every NPU port, including PFC pause and discard counters; (2) verify MTU 4200 and PFC/ECN are consistent end-to-end; (3) validate the ranktable rank-to-device-to-NIC-to-IP mapping and confirm both NICs are on the intended rail/leaf; (4) run hccl_tools all_reduce_perf at 32 and 64 ranks, and compare intra-node, inter-node, and single-rail results; (5) collect per-rank step times to find outliers. If the 64-rank bandwidth test is clean and all ranks are uniformly slow, revisit compute and collective algorithms; otherwise fix the fabric before adding nodes.
2. Specify the remediation and acceptance criteria. Include what you would change in ranktable generation, NIC/switch configuration, and HCCL settings, and how you would prove the fix.
Read the worked answer
Remediation: normalize MTU to 4200 on all nodes and switch ports; enable PFC/ECN consistently on every RoCE port carrying HCCL traffic; move both NICs per server onto the same rail/leaf pairing so the topology is symmetric; regenerate the ranktable from actual device and IP inventory with hccl_tools.py rather than hand-editing, and validate every rank entry. Pin HCCL traffic to the intended NICs and use multi-rail settings only where the rail mapping is verified. Do not add nodes until the fabric is symmetric.
Acceptance criteria: 64-rank all_reduce_perf per-rank bus bandwidth within about 10% of the 32-rank result; no sustained PFC pause storms or port discards; step time at 8 nodes no worse than about 2.2 s for the same global batch; AI Core utilization above 75%; and per-rank step-time spread under 5%. Prove it with a before/after bandwidth test, a short 50-step training run with profiler traces, and one full checkpoint save/restore to confirm the ranktable remains correct after restart. If these pass but scaling is still poor, then investigate collective algorithm choice, gradient accumulation, or model-parallel partitioning.