RoCEv2 Incast Slowdown on a Spectrum-X Ethernet AI Fabric
A 64-node GPU cluster uses a 400 Gb/s Ethernet AI fabric with Spectrum-X switches, BlueField-3 DPUs, and RoCEv2. Each node has eight GPUs, and the fabric uses a lossless priority for storage and training traffic. During all-to-all phases of a mixture-of-experts model, several jobs slow down by 35%, while point-to-point bandwidth tests are clean. Telemetry shows PFC pause frames rising sharply on uplinks during the slow phases, some ports reporting discard counts, and ECN-marked packets increasing. The application team says the model's traffic pattern is bursty and that the same model ran better on an older fabric. The network team has not changed ECN thresholds since the fabric was installed, and DSCP marking is applied by the host but not verified end to end.
1. How would you determine whether the problem is congestion control, lossless configuration, or application traffic pattern?
Read the worked answer
Separate the three by controlled measurement. First, reproduce the all-to-all pattern with a synthetic benchmark at the same message sizes and fan-in, then compare it with the point-to-point test. If the synthetic all-to-all reproduces the slowdown, the pattern is a major factor; if not, the application's timing or synchronization is involved. Second, capture per-priority counters: PFC pause duration and pause storms, ECN mark rate, buffer occupancy, and discards. Rising PFC pauses with ECN marks suggests the congestion control loop is not reacting early enough, so the lossless queue is absorbing bursts and then pausing. Discards on a lossless priority are especially serious because they indicate the buffer or PFC configuration is not protecting the queue.
Third, verify DSCP marking end to end from host to switch and back, because a mismatch can put RoCE traffic into the wrong queue and bypass the intended ECN/PFC treatment. Check MTU consistency and whether the DPU and switch agree on the lossless priority. The older-fabric comparison is useful only if you confirm the same model, message sizes, and collective library settings; otherwise it is anecdotal.
2. What tuning and validation plan would you propose, and what would you refuse to change without evidence?
Read the worked answer
Tune the congestion control loop before changing queue assignments. Verify that ECN is enabled on the correct priorities and that the marking thresholds match the switch buffer and the fabric's expected burst size. If ECN marks appear only after PFC has already started, lower the ECN minimum threshold so senders back off earlier, and validate that the sender reacts to congestion notifications. Ensure PFC is enabled only on the intended lossless priority and that PFC watchdog is configured to detect a pause storm and recover the port. Check that adaptive routing is enabled consistently and that the hashing does not pin an all-to-all flow set to one uplink.
I would refuse to disable PFC entirely on a RoCEv2 lossless fabric without evidence, because that can convert congestion into packet loss and trigger severe retransmission. I would also refuse to raise ECN thresholds just to make marks disappear. Validate with a canary job: run the synthetic all-to-all and the real model on a small set of nodes, then compare p99 completion time, PFC pause duration, ECN mark rate, and discards. Only scale to all 64 nodes after the canary shows no pause storms and no discards.