AI Training Cluster Storage Choice: Alletra 9000 versus ClusterStor E1000
An oil-and-gas research group is building an AI training cluster: 16 HPE Apollo 6500 Gen10 Plus nodes with 8 GPUs each, 24 HPE ProLiant DL385 Gen11 storage nodes, and a 400 GbE RoCE fabric. The training dataset is 180 TB of small files averaging 2 MB and 12 TB of large checkpoints. Each training job reads the dataset 4 times per epoch, writes a 2 TB checkpoint every 20 minutes, and must complete 30 epochs in 10 days. The team proposes a single HPE Alletra 9000 block array with 200 TB usable, exported over NVMe-oF, and NFS for checkpoints. They expect 60 GB/s aggregate read and 12 GB/s write. Current NFS tests show 9 GB/s read and metadata latency of 40 ms. The budget allows one additional platform. They need to choose between scaling the Alletra 9000 and adding an HPE Cray ClusterStor E1000.
1. Assess whether the proposed Alletra 9000 plus NFS design can meet the read, write, and metadata requirements.
Read the worked answer
The proposal conflates block performance with file-serving capability. Alletra 9000 over NVMe-oF can deliver high IOPS and low latency to a small number of clients, but the workload is a shared file dataset of about 90 million 2 MB objects, read four times per epoch by 16 GPU nodes. The NFS test already shows 9 GB/s read and 40 ms metadata latency, far below the 60 GB/s target. The metadata path, not just the block array, is the bottleneck. Checkpoint writes are bursty: 2 TB every 20 minutes averages about 1.7 GB/s, but the 12 GB/s target implies a short burst, so the platform must absorb that without starving training reads. A single block array exported through NFS also creates a scaling and failure-domain risk. The design is unlikely to meet the requirement without a parallel file system and a metadata architecture sized for the file count.
2. Recommend the platform and architecture, including capacity and protocol choices.
Read the worked answer
Add an HPE Cray ClusterStor E1000 running Lustre and keep the Alletra 9000 for block workloads or validated checkpoint volumes, rather than forcing NFS through it. Lustre separates metadata and object storage, scales metadata performance across MDS nodes, and lets clients stripe reads across many OSTs. Size raw capacity for 180 TB dataset plus 12 TB checkpoints, RAID and sparing overhead, and at least 20-30% headroom; a 300 TB raw starting point is more realistic than 200 TB usable. Use NVMe or SSD metadata targets and enough OSS/OST capacity to sustain 60 GB/s read and 12 GB/s checkpoint bursts over 400 GbE RoCE. Configure progressive file layouts: large stripe counts and large stripe size for checkpoints, smaller but still parallel layouts for the 2 MB dataset. If checkpoints require block semantics, validate Alletra 9000 NVMe-oF as a separate tier, but do not use NFS as the primary AI data path.
3. What performance validation and operational risks should be addressed before committing to the design?
Read the worked answer
Benchmark with representative tools: IOR or fio for large sequential reads and checkpoint bursts, and MDtest for metadata operations across the 90 million files. Run from all 16 GPU nodes concurrently and confirm aggregate 60 GB/s read and 12 GB/s write bursts, not single-client results. Monitor RoCE for PFC/ECN drops, MTU consistency, and congestion; a lossy fabric will not sustain the target. Validate Lustre OST balance, MDS CPU, client striping, and job-scheduler behavior when many jobs open the same files. Test failure of an OSS, MDS, and client link, and confirm checkpoint restart without data corruption. The main risk is small-file metadata overhead: if MDtest cannot meet the epoch schedule, consider dataset sharding or a different layout before buying more capacity. Document a rollback path and benchmark the Alletra 9000 checkpoint tier separately if it remains in scope.