Deferred Maintenance Catches Up With the Batteries
A 2 MW data centre runs four UPS modules, three 1.5 MW diesel generators, and VRLA battery strings now four years old. The battery room has drifted to 28 °C because a CRAH serving it was repurposed during an expansion. Last impedance test was 18 months ago and one string already showed a 12% deviation. Generators are exercised monthly for 30 minutes at roughly 30% load; no load-bank test has been run since commissioning. Summer humidity in the cold aisle has reached 65% RH with humidifiers offline since a parts delay. The CMMS shows 38% of planned maintenance overdue. Staffing is two technicians per shift; neither holds a high-voltage switching authorisation. During a 12-minute utility outage last week, measured UPS autonomy was four minutes against a ten-minute design, and one generator failed to start on first attempt before catching on the second.
1. Rank the corrective actions for the next 30 days, justifying each ranking in risk terms rather than by ease or cost.
Read the worked answer
Ranking logic: score each issue by (probability of failure) × (consequence during a utility outage or a fire/humidity event) ÷ (time to effective mitigation). Speed of mitigation matters because the exposure is live now.
- Generator start reliability - highest. A generator that failed once will fail again, and it is the only long-duration energy source. Immediate actions: replace starter batteries on all three sets, inspect fuel system and block heaters, then run a full-load acceptance test on each generator, not a 30% exercise. A 30% monthly run proves almost nothing about a 1.5 MW machine; wet stacking and unproven step-load capability are the real risks.
- Battery autonomy at 4 minutes against 10. This determines whether the site survives the window between outage and generator pickup. The 12% deviating string should be treated as failed now: replace it, then perform a discharge test on all strings to establish real autonomy. Interim compensating measure: reduce critical load or bring forward generator start, and document the reduced autonomy in the SLA risk log.
- High-voltage switching competence. Two technicians without HV authorisation means the site cannot safely execute a planned isolation or respond to certain faults. This is both a safety and an availability issue. Book an authorised-person course and, until competent staff are on shift, contract an authorised engineer on call with a defined response time.
- Humidity at 65% RH. Consequence is condensation and corrosion risk plus latent load on cooling, not an immediate outage. Restore humidifier parts and add temporary portable humidification if the room is genuinely dry; confirm whether the excursions are real or a sensor calibration artefact first.
- Temperature at 28 °C in the battery room. Every 10 K above 25 °C roughly halves VRLA design life, so this is a slow asset-destruction issue. Restore dedicated cooling to the room within 30 days; it is urgent but not an outage driver this week.
Cross-cutting: the 38% overdue PM backlog is the root cause, not a separate item. Freeze new project work until the backlog is below 10% and re-baseline the PM schedule against actual staffing.
2. The customer SLA promises 99.99% and there is no budget approval until next quarter. Define the compensating measures you would put in place and how you would communicate them.
Read the worked answer
Principle: when the design intent cannot be restored immediately, reduce the probability of the initiating event, shorten recovery, and make the residual risk explicit to the people who own it. Do not let an unfunded gap remain undocumented.
Compensating measures:
- Generator readiness: weekly no-load start plus a documented fuel, oil and coolant check; place spare starter batteries on site; contract the generator OEM for a 4-hour on-site response with a named engineer. Schedule the full-load test as the first spend of next quarter's budget.
- Battery: replace the deviating string under operational expenditure if capital is blocked; run a partial discharge test to quantify real autonomy; if autonomy remains below design, formally derate the published ride-through figure and adjust the load profile.
- Switching: contract an authorised HV engineer on a standby rota with a maximum 60-minute attendance, and prohibit live switching by unauthorised staff - the safety exposure is not negotiable against a budget cycle.
- Monitoring: raise alarm thresholds so a generator fail-to-start or a battery-room temperature excursion pages on-call immediately rather than appearing in a daily report.
- Humidity: verify with a calibrated handheld sensor before spending; if confirmed, deploy portable units and log the excursions.
Communication: send the customer a factual service risk notice - what was observed (4 minutes autonomy, one failed start), what is being done, what residual risk remains until which date, and what the customer can do (reduce non-critical load). Copy the account owner and record it in the risk register with an owner and a review date. The reputational damage comes from the customer discovering the gap during an outage, not from disclosing it now. If the 99.99% commitment cannot be supported by evidence, that conversation belongs in the notice, not in the next incident review.
3. The post-incident review finds no single technician error. Where should the review focus, and what outputs would make a real difference?
Read the worked answer
Focus on system conditions, not individuals. Every symptom here traces to a management decision made months earlier:
- The CRAH was repurposed during an expansion - a design change with no thermal re-assessment of the battery room. That is a change management failure, and the management-of-change process should require an environmental impact check for any plant reallocation.
- Humidifiers sat offline for a parts delay with no interim measure and no risk acceptance. That is a spares and risk-acceptance failure: critical spares need a stocking policy, and any deferral needs a signed acceptance with an expiry date.
- PM is 38% overdue with two staff per shift. That is a resourcing and planning failure measurable in advance; the CMMS data was telling the story for months.
- 30% generator exercises passed while real capability was unproven. That is a test-validity failure: the test measured the wrong thing.
- No HV authorisation on shift is a competence and succession failure that an audit should have caught.
Outputs that would change behaviour:
- A revised criticality register, so PM on life-safety and ride-through equipment cannot be deferred by the same rule as cosmetic tasks.
- A deferral gate: any PM overdue beyond its interval requires written risk acceptance by the site manager with an expiry and a compensating measure.
- Test standards rewritten to state what capability is being proven - full-load generator test annually, battery discharge test at defined intervals, with pass criteria.
- A change-impact checklist covering thermal, electrical and fire systems for every plant modification.
- A staffing and competence plan with a named deputy for HV switching and a training budget line.
The review should close with owners and dates, and be re-audited in 90 days. A report with no re-audit is a narrative, not a control.