AI/ML Storage and Data Management
This domain tests how you feed GPUs and protect training progress: choosing the right storage tier and protocol, sizing capacity and bandwidth, and integrating storage traffic into the Cisco network without damaging the lossless AI fabric.
Tiers and protocols
- Parallel file systems (scratch and hot datasets) deliver aggregate bandwidth by striping across many servers. They are the right answer for thousands of concurrent readers.
- NAS (NFSv3/v4.1, pNFS, NFS over RDMA) suits shared home directories and moderate datasets; default mount options often underperform.
- Object storage (S3-compatible) is the landing zone and dataset archive; it is high-throughput but high-latency.
- Block (FC, iSCSI, NVMe over Fabrics TCP or FC) serves databases and metadata stores. Cisco MDS 9000 handles FC and FC-NVMe; Nexus 9000 handles NVMe/TCP and NAS front-ends.
- GPUDirect Storage streams data from NVMe straight into GPU memory, removing host-memory copies.
Sizing a checkpoint
A 70B-parameter model in BF16 has about 140 GB of weights. Adam keeps two FP32 moments, adding about 560 GB, and gradients in BF16 add about 140 GB, so one checkpoint is roughly 840 GB. With optimizer state sharded across 64 GPUs, each GPU writes about 13 GB. If the storage delivers 200 GB/s aggregate, the checkpoint completes in about 5 seconds of pure write time; at 20 GB/s it takes about 45 seconds and starts to interfere with the next training iteration. That is why checkpoint bursts, not steady-state average, drive the bandwidth requirement.
Reads are usually large and sequential, so bandwidth and metadata operations per second matter more than random IOPS. Small-file workloads (millions of images) stress metadata servers, not the array.
Fitting storage into the fabric
Put storage traffic in its own QoS class and its own VLAN/VRF. Do not merge it into the same no-drop class as RoCEv2 unless you have budgeted headroom for both; a storage burst inside a PFC class can pause GPU traffic.
Traps that cost marks
- Sizing on average throughput and ignoring checkpoint bursts.
- Comparing IOPS when the workload is large sequential reads.
- Ignoring metadata performance for small-file datasets.
- Running storage and RoCEv2 in one lossless queue.