Calder Valley Water: A Stale Plan and a Generator That Would Not Start
Calder Valley Water Authority supplies 420,000 households and operates 96 pumping stations and a SCADA system on Windows Server 2012 R2. Its DR plan is a 210-page document last revised three years ago; it names a DR coordinator who left 18 months ago and lists 14 applications, four of which have been decommissioned. The alternate control room has a generator last load-tested in 2021; during a recent power outage the generator failed to start because its starter battery was flat. Staff restored pumping manually after 11 hours.
The regulator requires annual testing and evidence of continuous improvement. The plan has only ever been tested by a two-hour tabletop with six managers. You are the new resilience lead, and the operations director says 'we know what to do in a crisis; testing just creates risk'.
1. Diagnose the plan maintenance and quality failures and propose a maintenance regime with clear ownership.
Read the worked answer
The incident exposes a plan that has become a historical document rather than an operational tool.
Failures:
- No current owner: the named coordinator left 18 months ago and was not replaced.
- No change-trigger review: four decommissioned applications remain in the plan, showing no link to configuration or change management.
- No maintenance of supporting assets: the generator starter battery was not checked, and the last load test was three years ago.
- No exercise evidence: a single tabletop with six managers does not validate communications, manual operations or technology recovery.
- No corrective action loop: the 11-hour manual restoration was not analysed against a tolerable period.
Maintenance regime:
- Appoint a plan owner and two alternates, with defined responsibility in role descriptions.
- Review the plan annually and on triggers: organisational change, application change, site change, vendor change, regulatory change or a significant incident.
- Link the plan to the configuration management database so application and contact inventories update automatically or are reconciled quarterly.
- Set asset maintenance schedules, including generator load tests at the manufacturer's recommended interval and battery replacement.
- Track exercises and corrective actions in a register reviewed by management, with target dates and owners.
- Distribute controlled copies to the alternate control room and keep an offline copy accessible during a SCADA outage.
2. Design a 12-month progressive exercise programme and justify the sequence.
Read the worked answer
The programme should move from low-risk validation to controlled operational testing.
- Quarter 1 - walkthrough: process owners and control-room staff walk through the plan against a loss-of-primary-control-room scenario. Validate roles, contact details, application inventory and manual procedures. Output: corrected plan and a list of gaps.
- Quarter 2 - tabletop simulation: include operations, communications, legal, regulator liaison and customer contact. Inject loss of SCADA, staff absence and a failed generator. Test decision-making and escalation, not technology. Output: revised decision log and communication templates.
- Quarter 3 - component tests: generator load test, UPS test, historian failover, backup restore and network failover, each with acceptance criteria. These are low-impact and can be scheduled off-peak.
- Quarter 4 - functional exercise: relocate a limited team to the alternate control room, operate pumping manually or remotely for a defined period, and measure performance against the MTPD. Include a regulator notification drill.
After every exercise, hold an after-action review, record corrective actions with owners and dates, update the plan and report to management. The sequence builds confidence and evidence before a full operational test, which is important for a water utility where an uncontrolled exercise could affect supply.
3. How would you answer the operations director's objection that 'testing just creates risk'?
Read the worked answer
The objection contains a valid concern but reaches the wrong conclusion. Testing can create risk if it is uncontrolled. Not testing guarantees that weaknesses remain hidden until a real outage, when the consequences are uncontrolled.
The recent generator failure is the strongest evidence: a scheduled load test would probably have detected the flat starter battery before the outage. The 11-hour manual restoration also shows that the organisation can improvise, but not that it can meet a defined recovery objective.
My response would set out controls that make testing safe:
- Define scope, objectives, stop criteria and rollback before each exercise.
- Use change control and schedule off-peak where supply could be affected.
- Start with walkthroughs and tabletops that cannot affect operations.
- For operational tests, have experienced facilitators, safety observers and a clear abort authority.
- Keep the regulator informed where required and align with the safety case.
- Report results as evidence of control, not as a search for blame.
The director's real concern is operational stability. The programme should acknowledge that and show how testing reduces, rather than adds to, the risk of an unplanned failure.