Checkpoint Storm: Diagnosing a Storage Fabric That Collapses Every 15 Minutes
A 64-node training cluster writes a 1.2 TB checkpoint every 15 minutes. Each node has eight H100 GPUs and two 100G storage NICs; the parallel file system presents two 100G uplinks. During checkpoints, aggregate write throughput stalls at 6 GB/s, show interface reports millions of pause frames on priority 3, and hosts log TCP retransmits. The storage VLAN is also used for PXE boot and backup, MTU is 1500 on the switches but 9000 on the hosts, and ECN is disabled. The storage vendor guarantees 20 GB/s. Data scientists report that a checkpoint that should take two minutes takes eleven, and some jobs fail after three retries. The team wants a fix that does not require re-architecting the file system.
1. Separate the capacity problem from the congestion problem. What is the actual demand, and why is 6 GB/s not simply a tuning issue?
Read the worked answer
Compute the demand first. A 1.2 TB checkpoint every 15 minutes is 1.2 TB / 900 s = 1.33 GB/s per node if spread evenly. With 64 nodes writing simultaneously, aggregate demand is about 85 GB/s. The file system presents two 100G uplinks, about 25 GB/s theoretical and perhaps 20 GB/s practical, and the vendor guarantees 20 GB/s. Even perfectly tuned, the storage target is oversubscribed by roughly 4:1 against synchronized checkpoints.
That explains the stall at 6 GB/s only partly. The additional loss comes from congestion and errors: MTU 1500 on switches versus 9000 on hosts breaks large-frame transmission and causes drops or fragmentation; ECN is disabled, so PFC is the only backpressure mechanism; the storage VLAN is shared with PXE and backup, so unrelated traffic competes; and hashing two 100G links with few large flows can imbalance traffic. Pause frames on priority 3 then propagate, and TCP retransmits add load.
So there are two fixes: reduce or stagger demand, and repair the path. Tuning alone cannot make 85 GB/s fit into 20 GB/s. A defensible target is to stagger checkpoints so no more than 16 nodes write at once (about 21 GB/s), or add storage bandwidth. Until demand is shaped, every queue and buffer change is temporary.
2. Design the storage path and checkpoint policy. Be specific about isolation, MTU, flow control, and where buffering belongs.
Read the worked answer
Isolate storage traffic first. Put checkpoint and file-system traffic on dedicated VLANs or a dedicated storage fabric, separate from PXE, backup, and management. If a separate fabric is not possible, use strict QoS: classify storage to a no-drop priority, cap backup and PXE, and ensure the storage class cannot be starved by management traffic. Dedicated 400G uplinks from the leaf pair to the storage cluster would give roughly 50 GB/s and remove the 4:1 oversubscription.
Set MTU 9216 end to end on servers, leaf, spine, and storage ports; a mismatch is not a tuning detail, it is a fault. Enable ECN on the storage priority with thresholds tied to the actual buffer size, and keep PFC as a last resort rather than the primary control. With RoCE, ECN marking lets the sender reduce rate before PFC pauses the link. If TCP is used, ECN still helps, but the bigger lever is reducing synchronized incast.
Buffer belongs at the leaf facing the storage cluster, not spread evenly, because that is where many-to-one convergence occurs. Use per-priority buffering and monitor headroom. On the host side, write checkpoints to local NVMe first and flush asynchronously in a staggered schedule; this decouples GPU job progress from storage latency and gives retries a durable local copy. Keep the parallel file system, but add a checkpoint coordinator so ranks do not all flush at the same instant.
3. What validation and monitoring would prove the fix, and what would you refuse to accept as evidence?
Read the worked answer
Prove it with a production-like checkpoint, not a single-stream dd. Run a canary group of 16 nodes, then 32, then 64, each writing a full-size checkpoint while collecting per-node throughput, per-link PFC pause duration, ECN-marked packet counts, retransmits, and job pause time. Acceptance should be a checkpoint completion time and a GPU idle-time budget, for example under three minutes at 64 nodes with no job retry. Also test a storage uplink failure and a leaf failure to confirm the checkpoint can restart from local NVMe.
Monitor continuously: interface pause counters, queue drop counters, ECN marks, MTU mismatches, and storage client latency. Alert on pause duration, not only pause count, because a few long pauses hurt more than many short ones. Correlate with the scheduler so a checkpoint storm is visible as a job event.
Refuse to accept link utilization alone, a single-node benchmark, or a vendor guarantee as proof. Also refuse a fix that disables PFC without enabling ECN, or one that sets jumbo frames only on hosts. Those change symptoms without closing the demand or path problem.