Storage Configuration and Management
What this domain tests
Feeding accelerators fast enough to keep them busy: local NVMe scratch layout, parallel filesystem clients, GPUDirect Storage, checkpoint strategy, and the bandwidth arithmetic that tells you whether the pipeline can keep up.
Core rules
- Do the bandwidth math first. A node with eight GPUs each consuming 1-2 GB/s of samples needs 8-16 GB/s at the node, well beyond a single 100 Gb/s link (about 12.5 GB/s theoretical). This is why training clusters use multiple high-speed NICs per node.
- GPUDirect Storage. GDS moves data from NVMe or a supported parallel filesystem straight to GPU memory via the cuFile API, bypassing the CPU bounce buffer. It requires the
nvidia-fskernel module, a compatible filesystem, and validation with the bundledgdscheckandgdsiotools. - Local scratch. NVMe RAID 0 is normal for per-job scratch; RAID 5 rebuild cost and write amplification hurt training I/O more than the redundancy helps, since checkpoints belong on shared storage.
- Checkpointing. Write sharded, asynchronous checkpoints to a parallel filesystem (Lustre, BeeGFS, WEKA, GPFS) rather than serializing from rank 0.
Worked example
A job sits at 20 percent GPU utilization with high I/O wait, and the data loader reads 4 KB random blocks over NFS. Counting it out: 8 GPUs x 1.5 GB/s = 12 GB/s required, but the observed NFS throughput is under 1 GB/s, so the GPUs are starved by about 12x. Fixes in order of impact: convert the dataset to large sequential shards, place it on a parallel filesystem, and give each rank its own reader so reads parallelize.
Traps that cost marks
Claiming GDS is active without checking that nvidia-fs is loaded and the filesystem is supported; benchmarking with tiny block sizes that measure IOPS instead of streaming bandwidth; assuming NFS is adequate for multi-node training because it was adequate for one node.