Sizing a 1,024-GPU Training Hall Against a 1.2 MW Power Ceiling
A colocation provider has allocated a 900 m2 hall with 1.2 MW of usable critical power and 1.1 MW of cooling for a new AI training cluster. The data science team wants 1,024 NVIDIA H100 GPUs in 128 eight-GPU nodes, a 20 PB parallel file system, and 12 management servers. Each GPU node draws about 10 kW at the wall; storage and management add 90 kW. Jobs run for two to four weeks and are tightly synchronized. The business asks for 1:1 non-blocking GPU-to-GPU bandwidth but has approved only 60% of the optics budget for a full two-tier fabric. Existing racks are rated at 15 kW and the provider offers air cooling only. The WAN handoff is two 400G links. Design review is in two weeks.
1. Reconcile the power, cooling, and rack-density constraints. What deployment model do you recommend, and what must change before 128 nodes can be installed?
Read the worked answer
Start with the load: 128 nodes x 10 kW = 1,280 kW, plus 90 kW for storage and management, gives 1,370 kW of IT load. That is 170 kW above the 1,200 kW usable ceiling and well above the 1.1 MW cooling capacity. Rack density is worse: four nodes per rack is 40 kW, and even one node plus a ToR switch at about 11 kW consumes most of a 15 kW air-cooled rack, so a 128-node layout is physically impractical.
Recommend direct-to-chip liquid cooling with CDUs and 40-60 kW racks, four nodes per rack. Liquid cooling removes most GPU heat to the facility water loop and allows a realistic PUE near 1.15-1.25 instead of air-cooled 1.4+. Even then, 1,370 kW at PUE 1.2 is about 1,644 kW at the facility, so the hall cannot hold all 128 nodes at 10 kW.
Phase the build. Ninety-six nodes is 960 kW + 90 kW = 1,050 kW IT; at PUE 1.14 that is about 1,197 kW, just inside the 1.2 MW ceiling, leaving no headroom for growth. A safer first phase is 84 nodes (930 kW IT, about 1,060 kW facility), with the remaining 44 nodes added after a power upgrade or after measured power capping at 600-650 W per GPU is validated. Require per-rack power telemetry and a stranded-power calculation before approving the second phase; do not sign up to 128 nodes on the promise of future power.
2. Given the 60% optics budget, propose a fabric topology and oversubscription plan that preserves training efficiency. Show the port and optics arithmetic.
Read the worked answer
Assume one 400G NIC per GPU, which is common for H100-class systems: 128 nodes x 8 GPUs = 1,024 access ports. A fully non-blocking two-tier fabric needs 1,024 access optics plus 1,024 fabric optics, or 2,048 optics. The 60% budget is roughly 1,229 optics, so a full 1,024-GPU 1:1 build cannot be funded in one pass.
Build in a 64-node pod, which is 512 GPU ports. Within the pod, use a rail-optimized mapping: GPU i on every server connects to the leaf group for rail i. For 512 ports, use 16 leaves with 32 downlinks and 32 uplinks each, and 8 spines with 64 ports each. That is 24 switches, 512 access optics and 512 fabric optics, or 1,024 optics total, inside the 60% cap. Every GPU gets 400G access and the pod is 1:1 non-blocking.
Add the second 64-node pod later. Connect pods with a super-spine at 2:1, so intra-pod all-reduce and all-to-all remain non-blocking and only cross-pod traffic is oversubscribed. Keep the same rail index across pods so NCCL rank placement maps predictably to rails. This is better than halving access speed on all 1,024 GPUs: a 200G access design saves optics but permanently caps every collective. Recalculate optics after the second pod and negotiate the 60% cap against the phased schedule.
3. What operational, resiliency, and sustainability measures should be conditions of design acceptance?
Read the worked answer
Make acceptance measurable rather than a checklist. For resiliency, require no single spine failure to reduce pod bisection below 1:1, dual-homed storage, and a tested drain procedure that lets a leaf be upgraded without killing a multi-week job. The scheduler must checkpoint to local NVMe before any fabric maintenance; fabric redundancy does not protect against a bad change.
For operations, require streaming telemetry for per-link PFC, ECN, buffer utilization, and CRC errors, plus GPU utilization and job-scheduler state. Nexus Dashboard Insights and Intersight should be integrated so a fabric event can be correlated with a node or job event. Define alert thresholds before day one; otherwise the team will discover congestion only when a job slows.
For sustainability, require per-rack power metering, PUE reporting, CDU flow and leak detection, and a documented water-temperature design. Use measured GPU power to set capping policy rather than a blanket cap. Heat reuse is optional but should be evaluated with the facility team. Add a canary rack: burn in the first four nodes with a production collective and a checkpoint before filling the hall. Finally, require a decommissioning and optics-reuse plan, because 400G optics are a large recurring cost and a sustainability factor.