Checkpoint Stalls on an NFS-Backed Training Cluster
You support a 64-node, 512-H100 training cluster running a 175B-parameter model. Every 30 minutes the job writes a 2.4 TB checkpoint and must finish in under 5 minutes to avoid stalling the optimizer. Checkpoints currently go to an NFSv4.1 server with 24 NVMe drives and two bonded 100GbE links; each client also has two 100GbE links. During checkpointing, write throughput peaks at 1.1 GB/s, many ranks enter D state, nfsstat shows a large RPC backlog, and mountstats reports high metadata operation latency. The training team writes all ranks into one shared checkpoint file. You must calculate the required throughput, identify the limiting layer, and decide whether to tune NFS or change the checkpoint architecture before the next run.
1. Calculate the throughput target and assess whether the network, NFS server, or checkpoint access pattern is the limiting layer.
Read the worked answer
The target is 2.4 TB in 300 seconds, or 8 GB/s. Two bonded 100GbE links provide about 25 GB/s raw, roughly 20 GB/s practical, so the network alone is not the limit. The NFS server has 24 NVMe drives and should sustain more than 8 GB/s at the block layer, but NFSv4.1 with all ranks writing one shared file serializes metadata and locking, and client RPC slot exhaustion creates the observed backlog and D states. The high metadata latency and low aggregate throughput point to the access pattern and NFS protocol path, not raw link capacity. Tuning alone is unlikely to close a 7x gap.
2. What checkpoint architecture would you recommend, and how does it address the observed bottlenecks?
Read the worked answer
Move large checkpoints to a parallel file system or object store designed for concurrent writes, such as Lustre, GPFS, DAOS, or an S3-compatible store, and use a distributed checkpoint format that writes per-rank or per-shard files. This removes the single-file lock and metadata hotspot, spreads writes across many OSTs or object buckets, and lets ranks write in parallel. If NFS must remain, use multiple files striped across multiple NFS servers or pNFS, and avoid one shared file. Add asynchronous checkpointing or a local NVMe burst buffer so the optimizer does not block on the full 2.4 TB write. The architecture should be validated with a realistic 512-rank checkpoint, not a single-client copy.
3. What NFS and client tuning would you still apply, and how would you validate the result?
Read the worked answer
On clients, increase sunrpc.tcp_slot_table_entries and sunrpc.tcp_max_slot_table_entries, use nconnect with multiple connections, and verify rsize/wsize are at the maximum supported. On the server, confirm enough nfsd threads, sufficient memory for the working set, and that the export options match the workload. Measure with nfsiostat, mountstats, and nfsstat -s during a canary checkpoint. Validation must show checkpoint time under 5 minutes with all 512 ranks participating, no sustained D-state backlog, and stable metadata latency. If tuning does not reach 8 GB/s, keep the parallel-file-system design as the production path.