Small-File AI Training Bottleneck on a Single NFS Export
A research lab trains vision models on 60 TB of images, average file size 180 KB, roughly 340 million files. Eight GPU nodes each read the dataset in random order per epoch; aggregate peak read demand is about 4 GB/s. Storage is a single NFS server exporting 60 TB from a RAID 6 array of 16 x 8 TB NL-SAS drives over one 25 GbE link. Checkpoints are 400 GB every 20 minutes, written to the same NFS export. Jobs intermittently fail with 'stale file handle', and epoch time has grown from 40 to 95 minutes while GPU utilization averages 35%. The team proposes buying a faster NFS server with a 100 GbE NIC.
1. Diagnose why GPU utilization is low and epoch time increased, distinguishing storage causes from GPU causes.
Read the worked answer
The GPU utilization number points away from GPUs. Eight nodes at 35% utilization are waiting on data. The dataset is about 340 million files x 180 KB ≈ 61 TB, close to the stated 60 TB. Random-order reads mean the training job performs hundreds of millions of metadata operations per epoch against a single NFS server. A single 25 GbE port tops out at about 3.1 GB/s, below the 4 GB/s aggregate peak even before protocol overhead. The RAID 6 array of 16 x 8 TB NL-SAS can stream well but is poor at small random reads and metadata. Checkpoint writes of 400 GB every 20 minutes add bursts to the same export.
Epoch time rising from 40 to 95 minutes and 'stale file handle' errors are consistent with NFS server overload, export/lease problems, and retries, not GPU throttling. Buying a faster NFS server with 100 GbE would raise the network ceiling but leave the single metadata server and small-file problem in place. Measure metadata IOPS, per-node read latency, NFS retransmits, and checkpoint write time before purchasing.
2. Design a storage architecture for the training pipeline and explain the data layout.
Read the worked answer
Replace the single NFS server with a parallel or scale-out architecture designed for small files and many readers.
- Data layout: pack the 340 million images into sharded container files (WebDataset/TFRecord or large tar shards) of 100-500 MB, reducing metadata operations by orders of magnitude. Keep an index mapping image ID to shard and offset.
- Namespace: use a parallel file system with separate metadata servers (Lustre, GPFS, or a scale-out NAS) or object storage with an S3-compatible API. If the framework needs POSIX, choose parallel FS; if it can use S3, object storage plus a caching layer is simpler.
- Cache: put 4-8 NVMe drives per GPU node as a read cache, and prefetch shards ahead of the epoch.
- Network: 100 GbE or 200 GbE between nodes and storage, non-blocking.
- Checkpoints: separate fast NVMe pool or object bucket, not the dataset export.
This addresses both the 4 GB/s streaming need and the metadata storm; a faster single NFS server does not.
3. Propose a checkpoint strategy that reduces job restart time and storage load.
Read the worked answer
Checkpointing 400 GB every 20 minutes is about 0.33 GB/s average, but the write is a burst and it competes with training reads. Write checkpoints asynchronously: each rank writes its shard to local NVMe first, then a background process flushes to a parallel FS or object bucket with multi-part upload. Keep the last two checkpoints local on NVMe for fast restart and the last N in durable storage. Use sharded or distributed checkpoints so 8 nodes write in parallel rather than one process writing 400 GB. Align checkpoint frequency with epoch boundaries and loss tolerance; if 20 minutes of lost work is too much, use incremental or more frequent smaller checkpoints. Validate a restore from checkpoint on a spare node monthly. Separate the checkpoint network path or QoS it so it cannot starve dataset reads. This reduces restart time and removes the checkpoint burst from the training data path.