The 'nothing failed' transfer test
A 2.2 MW data centre commissioned in 2017 runs two 1,500 kW diesel generators in N+1 with 48 hours of on-site fuel, and eight strings of VRLA batteries supporting 2N UPS. Maintenance records show monthly generator runs of 25 minutes at no load, no load-bank test since commissioning, annual fuel polishing but no fuel quality test since 2019, and battery impedance trending with no discharge test since 2018. Design autonomy is 10 minutes at full load. A recent step-load test on two strings recorded 6 minutes 20 seconds to end voltage, and three blocks in string 5 show internal resistance 40 percent above the string mean. During last month's transfer test, generator G1 failed to start on the first attempt because of a flat starting battery; the load transferred to G2 after 14 seconds and the UPS carried the hall with no IT load lost. The operations manager describes the site as fully compliant because nothing failed.
1. Challenge the 'nothing failed' conclusion. What latent failures exist and how do they couple?
Read the worked answer
'Nothing failed' describes the outcome, not the state of the plant. What actually happened is that a demand event exposed a failure and a second defence absorbed it. That is a near miss, and it should be recorded as one.
The latent failures are: a generator starting battery that was not maintained to a tested condition, which is the classic latent defect because it only manifests at the moment of demand; no load-bank evidence that either generator can carry real load, since 25-minute no-load runs prove starting and running, not capacity; and battery autonomy measured at 6 minutes 20 seconds against a 10-minute design, which is roughly 63 percent.
They couple. The margin that absorbed the 14-second transfer was the UPS battery, and that margin is a third smaller than the design assumed. Two defences that look independent on the single-line diagram, generator start and battery ride-through, were both eroded at the same time. On a real utility outage with a slower start, the same site that 'lost nothing' this month could drop the hall. The correct record is two latent failures with a common trigger and a reduced safety margin, with corrective actions and dates.
2. Interpret the battery test result and set out the required action, including the standard or practice you rely on and what you do while replacement is pending.
Read the worked answer
The measured capacity is 6 minutes 20 seconds against a 10-minute design, about 63 percent of rating. Most operators apply a replacement criterion of around 80 percent of rated capacity, so this string is beyond end of useful life and cannot be relied on to bridge a transfer plus generator start.
The applicable guidance for VRLA maintenance, testing and replacement is IEEE 1188, with IEEE 450 covering vented lead-acid; use the capacity test at the specified discharge rate rather than impedance alone, because impedance trending is a screening tool and does not prove capacity. Actions: replace the three high-resistance blocks in string 5 and retest the string; a string performs to its weakest block, so if the retest still fails acceptance, replace the string as a unit and do not mix new blocks with aged ones without an engineering assessment. Reinstate a periodic discharge test programme, typically annual, instead of impedance-only trending.
While replacement is pending, document a formal risk acceptance, reduce the critical load on the affected UPS where possible, and check that the generator start sequence and starting batteries are proven so the battery is never the only thing standing between the load and an outage.
3. Assess the generator test regime. Why are monthly no-load runs and no load bank inadequate, and what would you change?
Read the worked answer
A no-load run tests the starting system and confirms the engine runs. It does not test the two things that matter in an outage: whether the set can carry the site load, and whether the fuel system can deliver at that rate. It also has a side effect. Diesel sets run light for long periods suffer wet stacking, unburnt fuel and carbon deposits in the exhaust and turbo, which degrades performance over time; the longer the site runs at no load, the more the set degrades while appearing to be maintained.
The change is to run monthly tests at load, at least 30 percent of rating where the load profile permits, and to perform an annual load-bank test at the site's actual load profile, including a step load that mimics the real transfer, per the manufacturer's guidance. Add a fuel quality test, since 48 hours of stored fuel with no quality check since 2019 is an unverified dependency; microbial growth, water and particulate all cause generator failure at the worst moment. Finally, put generator starting batteries on a maintenance regime with periodic load testing and charger verification, because that was the actual failure observed.