PFC Storms and ECN Policy in a 400G Two-Tier RoCE Fabric
A two-tier 400G RoCEv2 fabric carries distributed training between 64 leaf switches and 32 spines. PFC is enabled on priority 3 and ECN is configured, but after a new all-to-all workload is introduced, PFC pause frames propagate across the fabric and PFC watchdog shuts several leaf ports. show interface priority-flow-control shows pause storms on links that carry no RoCE flows, and application throughput collapses from 380G to 90G per host. The fabric uses 64 MB shared buffers per leaf, 400G downlinks, and 400G uplinks at 2:1 oversubscription. The operations team has been told to 'just enable PFC everywhere' and to disable ECN because 'marking causes retransmits.' A change window is available this weekend.
1. Explain the failure mechanism. Why are pause frames appearing on links that carry no RoCE flows, and why is disabling ECN the wrong response?
Read the worked answer
PFC is hop-by-hop backpressure. When a leaf egress queue for priority 3 fills, it sends a pause to its upstream neighbor. That neighbor may be a spine, and the spine's buffer then fills because it cannot drain toward the congested leaf, so it pauses other leaves. This head-of-line blocking propagates until ports that carry no RoCE flows are pausing. The watchdog shuts ports because the pause condition persists beyond its timer, which is a symptom of a buffer and policy problem, not a reason to disable the watchdog.
Disabling ECN makes it worse. ECN marking is the early signal that lets DCQCN or the transport reduce rate before buffers fill. Without ECN, the only feedback is PFC or packet drop, so congestion reaches the pause threshold faster and pauses last longer. ECN marking does not cause retransmits; retransmits come from drops, corruption, or timeouts. If the team sees retransmits, the causes to investigate are buffer exhaustion, MTU, or misclassified traffic, not the ECN mark itself.
The 2:1 uplink oversubscription concentrates all-to-all traffic at the spine. With 64 MB shared buffers and a new incast pattern, the fabric is being asked to absorb bursts that should be spread or rate-controlled.
2. Write the change plan for the weekend window. Include classification, ECN and PFC settings, buffer policy, and the order of operations.
Read the worked answer
Do not enable PFC everywhere. First classify: mark RoCEv2 traffic to a single no-drop priority, for example COS 3 with DSCP 26, and make sure all other traffic uses lossy priorities. Enable PFC only on that priority, on fabric and host-facing ports, and leave it disabled on priorities that carry management, storage, or best-effort traffic. This alone stops unrelated traffic from inheriting pause behavior.
Set ECN on the RoCE priority on every port. Use a minimum threshold low enough to mark before the shared buffer is half full and a maximum that forces marking aggressively before PFC. Because the platform shares 64 MB across ports, calculate per-port headroom and verify with the actual buffer counters rather than copying a threshold from another switch model. Keep PFC watchdog enabled with a sane timer; do not disable it to hide the problem. If the platform supports per-queue pause, use it to limit the blast radius.
Order of operations: apply classification and verify counters; apply ECN and verify marks increase without drops; apply PFC only on the no-drop class; then run a canary all-to-all on four hosts and scale to 16. Have a rollback that removes PFC from the class and restores the prior QoS policy. The 2:1 uplink oversubscription may still require rate limiting or a fabric upgrade; do not promise a full fix from QoS alone.
3. What telemetry and acceptance criteria would you use to decide whether the fabric is stable after the change?
Read the worked answer
Use a small set of leading indicators. Per-interface and per-queue: PFC pause duration and count, ECN-marked and CE-marked packet counts, buffer utilization high-water marks, and drop counters. Per-flow or per-job: achieved all-reduce bandwidth, tail latency, and retransmit rate. Correlate with the scheduler so an all-to-all job is visible at the same time as the counters.
Acceptance should be a sustained workload test, not a ping. Run the production all-to-all for at least an hour at expected scale. Require zero watchdog events, PFC pause durations below an agreed threshold on fabric links, no pause propagation to non-RoCE priorities, and throughput within an agreed percentage of the 380G baseline. If pause duration is still high but drops are zero, the fabric is not stable; it is surviving on backpressure.
Telemetry should be streaming, not polled every five minutes, because PFC events are microbursts. Use Nexus Dashboard or an equivalent collector to trend buffer and ECN data and to alert on watchdog events, pause duration, and sustained ECN marking above a percentage of packets. Review the thresholds after the first production job; a threshold that never fires is not a control.