40 kW Racks in a 2011 Vintage Hall
An operator runs a 1,200 m² legacy hall designed in 2011 for 500 W/m² and 5 kW per rack. It has a 600 mm raised floor, perimeter CRAHs providing 350 kW of sensible cooling at 24 °C supply, hot-aisle return without containment, and 2 × 200 kVA UPS modules in parallel (N+1) currently at 45% load. A business unit wants eight racks of GPU training nodes at 40 kW each (320 kW IT) plus 60 kW of storage, live within nine months. The landlord forbids structural penetrations; the electrical room has no spare breaker positions above 250 A; the only water available is a 40 mm domestic supply line. Existing tenants hold an availability SLA with penalties and there is a change freeze on the shared UPS. Finance has approved a cooling budget of €400k and wants deployment in two phases.
1. Determine whether air cooling can support 40 kW racks in this hall, and set out a technically defensible cooling architecture with the supporting numbers.
Read the worked answer
Verdict: air cooling alone is not feasible, and the arithmetic proves it.
At a 12 K air temperature rise, 40 kW requires 40 / (1.2 kg/m³ × 1.005 kJ/kg·K × 12 K) ≈ 2.76 m³/s, i.e. roughly 9,950 m³/h per rack. A 600 mm perforated tile at an aggressive 3 m/s face velocity delivers only about 3,900 m³/h, so each rack would need ~2.5 tiles at velocities the shared plenum cannot sustain - and eight racks would demand ~80,000 m³/h against a CRAH fleet rated at 350 kW total, most of which is already serving tenants. Hot-aisle return without containment also means supply air short-circuits, so usable capacity is well below nameplate.
Recommended architecture: direct-to-chip liquid cooling with a targeted air residual.
- DLC captures 70–80% of server heat. At 75%, liquid load ≈ 240 kW and residual air load ≈ 80 kW (10 kW/rack plus storage), which the existing CRAHs and new containment can plausibly absorb. Gap analysis must confirm the residual against real free capacity, not nameplate.
- The 40 mm line is a fatal constraint for once-through water. At 1.5 m/s a 40 mm pipe carries only ~1.9 l/s; a 240 kW loop at 6 K ΔT needs ~9.6 l/s. Therefore the design must be a closed facility-water loop with dry coolers, using the domestic line for makeup only. Confirm dry-cooler footprint and noise on the roof, since penetrations are barred.
- Rear-door heat exchangers are the fallback but cap out around 25–30 kW/rack and still reject to room air, so they cannot carry 40 kW alone.
- CDU and dry-cooler weight may exceed the 500 W/m² floor allowance; place plant on grade or verify structural loading before committing.
The judgement point is that the cooling budget cannot fix an airflow physics limit - the solution must change the heat transport medium, not add more air.
2. Assess the electrical capacity and redundancy position for the 380 kW IT addition, and state what must be true before phase 1 is energised.
Read the worked answer
Capacity check. 380 kW of IT at ~0.98 power factor is roughly 388 kVA. The UPS block is 400 kVA total in N+1, already carrying ~180 kVA (45%), leaving ~220 kVA of usable capacity - short by ~170 kVA before any cooling load is added, and before N+1 is preserved. Cooling, CDUs and pumps add roughly 30% more again.
Redundancy check. Loading a 2 × 200 kVA N+1 system to ~95% destroys the redundancy value: a single module failure would transfer load to bypass. The change freeze exists precisely to protect existing tenants' SLA, so any work on the shared UPS must go through change advisory approval with a documented MOP and rollback.
What must be true before energising:
- A separate UPS (or a new parallel block) sized for at least the phase 1 IT load at ≤80% utilisation, fed from new switchgear - the 250 A spare position at 400 V gives only ~173 kVA and is insufficient on its own.
- Generator and ATS capacity re-verified against the new total, including cooling plant, with a load test rather than a nameplate review.
- Cooling and electrical capacity released in matching increments - never energise IT before the heat rejection path is commissioned.
- A written SLA impact assessment signed off by the account owners of the existing tenants.
The failure mode to avoid is accepting a "temporary" non-redundant feed that quietly becomes permanent. If new switchgear cannot be delivered inside nine months, the honest recommendation is to phase GPUs at a lower density (e.g. 20 kW/rack) and re-plan rather than erode N+1.
3. Finance wants a two-phase plan it can hold you to. Outline the phase gates and the risks you would put in writing.
Read the worked answer
Phase 1 (months 0–5): prove the concept at reduced scale. Install four racks at 25–30 kW with DLC on a closed dry-cooler loop, new containment, and a dedicated UPS feed. Gate criteria, all measured not asserted: stable supply temperatures under full synthetic load for 72 hours; leak-detection and failover tested; PUE measured per EN 50600-4-2 for 30 days; CDU and dry-cooler redundancy demonstrated; no SLA breach attributable to the project.
Phase 2 (months 5–9): scale to 40 kW and add storage. Only proceeds if phase 1 gate data supports the full heat load and the second UPS/switchgear is commissioned. Storage goes on the air-cooled residual, not on the liquid loop, unless the vendor confirms cold-plate compatibility.
Risks to state in writing:
- Stranded asset risk: if phase 1 shows the residual air load is higher than modelled, the remaining racks cannot be populated without more CRAH capacity and the approved budget will not cover it.
- Schedule risk: switchgear and CDU lead times, not the GPUs, drive the nine-month date.
- Water and structure: dry-cooler siting and floor loading are outside your control and sit with the landlord; a refusal resets the design.
- Tenant risk: any unplanned event on the shared UPS converts a capacity project into an SLA liability.
The defensible message to Finance is a conditional two-phase approval with measurable gates, not a fixed delivery promise - the cooling budget alone does not buy the electrical headroom the load actually needs.