Sizing a Multi-Node Training Cluster for a Fixed Budget
A research lab has 8 DGX-class nodes, each with 8 GPUs and 640 GB of GPU memory total per node, connected by a non-blocking InfiniBand fabric. They want to train a 70B-parameter dense transformer on 4 trillion tokens within a 6-week window. Their data pipeline currently sustains 1.2 TB/s of host-to-device throughput per node, but their storage backend can only deliver 400 GB/s aggregate. During a pilot run on 2 nodes, they observe 78% GPU utilization and scaling efficiency of 0.91 when going from 1 to 2 nodes. A vendor proposes doubling node count to 16 and adding a second storage tier. The lab's power budget allows only 12 nodes to run concurrently at full draw. They ask whether to buy more nodes, more storage, or optimize the pipeline first.
1. How should the lab decide between adding compute, adding storage, and optimizing the data pipeline? What measurements would settle it?
Read the worked answer
The decision hinges on identifying the binding constraint rather than assuming compute is limiting. Three signals matter here.
First, the pilot shows 78% GPU utilization on 2 nodes with 0.91 scaling efficiency. That means roughly 22% of GPU time is already idle at small scale, before fabric contention and storage pressure grow. Idle time at 2 nodes usually indicates input starvation or synchronization overhead, not insufficient FLOPs. Adding nodes to a pipeline that already leaves 22% of the GPU idle tends to amplify the problem, because each additional node adds demand on the same 400 GB/s storage path.
Second, compare required versus available data bandwidth. A 70B dense model trained on 4T tokens implies a large per-step token batch; the checkpoint and shard loading during restarts also matters. If aggregate storage delivers 400 GB/s and each node can consume 1.2 TB/s host-to-device, then even a handful of nodes can saturate storage. The lab should measure actual bytes read per training step and compare with the storage ceiling. If reads per step exceed what storage can supply within the step time, storage is the constraint.
Third, run a controlled experiment: keep node count fixed at 2, vary only the data source (local NVMe cache versus network storage), and compare step time and GPU utilization. If local caching raises utilization from 78% toward 90%+, the pipeline is the bottleneck.
Recommended sequence: optimize the data pipeline and add caching first, since it is cheapest and the pilot already shows headroom. Then re-measure scaling efficiency at 4 nodes. Buy additional nodes only if utilization stays high and scaling efficiency remains above roughly 0.85. The 12-node power ceiling means the 16-node proposal is not executable at full draw anyway, so the vendor's plan needs a phased justification.
2. The lab proposes a 16-node cluster despite a 12-node power limit. What operational and architectural risks should they flag?
Read the worked answer
Several risks deserve explicit treatment.
Power and thermal headroom: a 16-node cluster that cannot run at full draw is effectively a 12-node cluster with 4 idle nodes. That wastes capital and rack space. The lab should either right-size to 12 nodes or design power capping so that all 16 nodes run at reduced clocks. Power capping changes performance characteristics and must be validated against the 6-week deadline.
Fabric and topology: scaling from 8 to 16 nodes changes the communication pattern. All-reduce and all-gather operations grow in message size and fan-out. The lab should verify that the non-blocking fabric claim holds at 16 nodes for the collective sizes used in training, not just at 8.
Storage contention: doubling nodes doubles concurrent readers. With 400 GB/s aggregate, more nodes means more contention and longer stalls unless the second storage tier is in place and benchmarked before the expansion.
Checkpointing: larger clusters write larger distributed checkpoints. Checkpoint write time can dominate if storage is slow, and a 6-week window has little slack for restart overhead.
Scheduling and failure domain: 16 nodes means more frequent hardware failures. The lab needs a job scheduler that can tolerate node loss and resume from checkpoint quickly.
Finally, the deadline risk: the pilot's 0.91 efficiency at 2 nodes does not predict 16-node efficiency. The lab should require a 4-node and 8-node scaling study before committing budget, and treat the 16-node proposal as contingent on measured efficiency.