Twelve HGX Nodes Into an Air-Cooled Colocation Hall
A colocation hall has 24 racks designed for 8 kW each, with in-row coolers rated at 12 kW per rack and room CRAC capacity of 350 kW total. Current IT load is 180 kW at a PUE of 1.55, with 22 C supply air. The company is installing 12 HGX H100 nodes, each 8U and rated at 10.2 kW maximum, and the original plan puts three nodes per rack, yielding roughly 30.6 kW per rack. Floor tiles are standard perforated tiles, and there is no liquid infrastructure in the hall. The provider offers rear-door heat exchangers and a direct-to-chip liquid option with a CDU, but both require changes to the rack layout and facility water. The project deadline is eight weeks, and the business wants all 12 nodes online at once.
1. Calculate the rack and room-level implications of the three-nodes-per-rack plan and identify the first hard constraint the design hits.
Read the worked answer
At 10.2 kW per node, three nodes per rack is 30.6 kW of IT load, well above the 12 kW in-row cooler rating and far above the 8 kW design density. The room total becomes 180 + 122.4 = 302.4 kW of IT load, which is below the 350 kW CRAC capacity, but at PUE 1.55 the facility load is about 469 kW, so the CRAC number is not the binding constraint. The first hard constraint is rack-level heat removal: 30.6 kW cannot be handled by 12 kW in-row coolers or standard perforated tiles at 22 C supply.
Even spreading the nodes to four racks at three per rack does not solve it, because each rack still exceeds 30 kW. Six nodes per rack would be 61.2 kW and is worse. The hall must either add liquid cooling or rear-door heat exchangers, or reduce nodes per rack and accept stranded space, which still may not meet 12 kW per rack. The room power headroom exists, but the cooling distribution and rack density do not.
2. Compare three deployment options and recommend one, with explicit trade-offs.
Read the worked answer
Option A is rear-door heat exchangers with the existing air infrastructure. It is faster to deploy and avoids facility water in the rack, but it still rejects heat to the room, so the CRAC load and PUE remain high, and it may not handle 30 kW per rack reliably at 22 C supply. Option B is direct-to-chip liquid cooling with a CDU. It handles the 10.2 kW node heat effectively and can support four nodes per rack at about 40.8 kW, but it requires facility water, leak detection, service procedures, and a longer commissioning path. Option C is to deploy fewer nodes per rack on air, for example one node per rack, which is operationally simple but consumes 12 racks and wastes space; it also may not meet the eight-week deadline if rack space is constrained.
Recommend Option B for four racks of three nodes each, or three racks of four nodes if floor loading and CDU capacity allow. Keep the room CRACs for the residual heat and set a PUE target. The trade-off is schedule and complexity, but Option A likely fails the thermal requirement, and Option C defeats the density and deadline goals. Make the recommendation conditional on a heat-load and water-quality review.
3. What monitoring and operational safeguards would you require before accepting the design?
Read the worked answer
Require per-node inlet and outlet temperatures, GPU and CPU throttle events, CDU supply and return temperatures, differential pressure, flow rate, and leak detection integrated into the DCIM or monitoring system. Set alarms well below the throttle threshold and test them by simulation. For rear-door or air options, require rack inlet temperature mapping at the top of the rack, because hot spots appear there first. For liquid, require quick-disconnect leak testing, drip trays, and a documented isolation procedure that lets one node be serviced without draining the loop.
Also validate at load, not at idle: run a sustained GPU burn-in and a real training job for at least 24 hours while recording power, temperature, and clock behavior. Confirm that a single CDU or cooler failure does not cascade into a thermal shutdown, and define a degraded mode with reduced node count. Acceptance should include a power capacity check at the breaker and PDU level, not only the room total.