Heatwave chiller loss and open containment
At 14:30 during a 41°C heatwave, a 600 kW IT hall loses one of three 350 kW chillers. The remaining two are at 95% load; chilled-water supply rises from 12°C to 17°C. Six CRAH units (five running, N+1) supply 18°C air, but rack inlet temperatures at the top of three high-density racks reach 31°C. Two hot-aisle containment doors are propped open because a contractor is pulling fibre. UPS load is 470 kW; UPS rooms are 29°C. The BMS shows one CRAH on alarm but still running. The site has 10-minute battery runtime at current load and a 15-minute generator start sequence. The incident commander wants to shut down non-critical customer racks immediately; the account team objects. The SLA allows 26°C inlet but has no heatwave derating clause.
1. What immediate incident decisions should the incident commander take, and how should customer shutdown be decided?
Read the worked answer
Declare a major incident and move to emergency response. First close the propped containment doors and stop fibre work; open containment is destroying the hot/cold separation and is likely why top-of-rack inlets hit 31°C. Do not shut down customer racks immediately unless temperatures exceed the customer's contractual limit or equipment protection thresholds; 31°C is above the 26°C SLA but not necessarily an equipment emergency. Instead: reduce non-critical load, start the spare chiller if available, lower CRAH setpoints cautiously to avoid coil condensation, and deploy portable cooling to the three hot racks. Put the CRAH on alarm in local control and verify airflow. Notify customers of the SLA excursion and the reason. If a rack inlet exceeds 35°C or the UPS room exceeds 30°C, execute a controlled shutdown of non-critical racks in a pre-agreed order. The account team does not own the safety decision; the incident commander does, with customer notification. Start the generator and confirm it is carrying load before any further risk.
2. Analyse the root causes and the design and operational changes needed.
Read the worked answer
Root causes: cooling capacity margin was calculated at design, not at 41°C; N+1 CRAH redundancy does not help when chilled water is short; open containment created a bypass path; and no heatwave procedure existed. The chiller loss left two units at 95% with 17°C supply, reducing CRAH capacity. Corrective actions: add chiller capacity or a thermal storage/adiabatic trim cooler; review CRAH redundancy at design wet-bulb; install door alarms and a permit system for containment; add rack-level inlet monitoring at top and bottom; write a heatwave playbook with trigger temperatures and load-shedding steps. Also check the BMS alarm: a CRAH on alarm but running may be on reduced capacity. The SLA needs a derating clause or a separate excursion protocol, but do not use it to excuse poor containment discipline. Test the playbook in a dry run before next summer.
3. What should be reported to the customer and to the board?
Read the worked answer
Customer: timeline, maximum inlet temperature and duration per rack, which racks exceeded the SLA, the cause (chiller loss plus open containment), immediate fixes, and the corrective-action dates. Avoid blaming the contractor in the customer letter until the investigation is complete. Board: this is a near-miss, not just an SLA credit. Report the capacity margin at 41°C, the open-door permit failure, the 10-minute battery runtime against a 15-minute generator sequence, and the cost of the fix. Recommend a resilience review across the portfolio: are other halls relying on N+1 CRAH with a single chilled-water loop? Set a 90-day gate: door alarms, heatwave playbook, chiller capacity study, and a witnessed failover test. If the board does not fund the chiller and containment fix, it should accept a lower published capacity or a heatwave exclusion-not leave the operational risk unowned.