Why passing DR tests produced a nine-hour outage
Arden Utilities has run an annual disaster recovery test for three years. Each test involved moving its billing platform to a secondary data centre at 2am on a Sunday, with a small IT team following a simplified runbook. All three tests passed with recovery in under two hours, and the board has been told the service is highly resilient. Last month a fibre cut combined with a failed network card caused a nine-hour billing outage. The incident review found that the test had excluded peak transaction volumes, customer service staff, the payment provider and the identity system; the runbook omitted three manual steps that only one engineer knew; and 40% of previous test action items were still open. The regulator has asked for a revised testing strategy. The COO wants to increase test frequency to twice a year and keep the same scenario. The head of resilience believes the test design, not frequency, is the problem.
1. Diagnose why the tests provided false assurance and propose a revised testing strategy.
Read the worked answer
The tests were valid for a narrow question-can a small IT team move billing to a secondary data centre at 2am?-but not for the question the board thought was being answered: can the billing service continue through a severe but plausible event at peak? The test excluded peak volumes, customer service, the payment provider and identity, and used a simplified runbook. It therefore tested component failover, not service resilience. The 40% open actions show the lessons-learned loop was not closed.
Revise the strategy around validity, coverage and learning:
- Scenario-led: severe but plausible events, including fibre cut plus hardware failure, cyber, third-party outage and loss of key people.
- End-to-end: include business staff, payment provider, identity and customer communications; test at peak volume and with degraded manual fallback.
- Unknowns: inject failures not in the runbook; require documented procedures rather than one engineer's knowledge.
- Independent observation: measure actual time to restore and reconcile against tolerance.
- Action governance: owners, dates, funding; escalate repeat findings.
- Portfolio: annual scenario tests plus quarterly component and third-party tests, proportionate to service criticality.
Frequency is secondary. Twice-yearly repetition of the same unrealistic scenario would add cost, not assurance.
2. How should the firm handle the 40% open action items and the COO's proposal? What governance would you put in place?
Read the worked answer
Treat open actions as a risk backlog, not an administrative issue. First, classify by service impact and whether they are preventive, detective or recovery actions. Any action that would have prevented or shortened the nine-hour outage is critical and should be funded and closed before adding new test scenarios. Assign owners at service level, set dates, and report overdue actions to the board risk committee. Investigate why they remained open-capacity, unclear ownership, no funding-and fix the cause.
On the COO's proposal: increasing frequency without changing design would industrialise false assurance. A better response is to keep one annual full scenario test, add targeted tests after remediation, and use the same budget to improve realism. If the COO wants more frequent evidence, use smaller component tests and third-party tests between full exercises. Require a test design review and independent sign-off before the next full test. The board should approve the revised strategy and the critical action closure plan together.
3. What metrics should the regulator see to evidence continual improvement?
Read the worked answer
The regulator should see evidence of effectiveness, not pass rates. Useful metrics:
- Coverage: percentage of important business services covered by at least one end-to-end test in the last 12 months; percentage including critical third parties.
- Performance: actual recovery time and data loss versus tolerance; percentage of tests that met their objectives; time to detect and contain.
- Remediation: open actions by age and severity; percentage overdue; repeat findings; mean time to close critical actions.
- Learning: number of tests that changed a procedure, runbook or investment decision; near-miss and incident trends.
- Third parties: percentage of critical suppliers tested or with validated exit plans.
- Governance: board and committee reviews; independent assurance coverage.
Avoid metrics such as number of tests completed, number of staff trained, or 100% pass rates, which can reward superficial testing. Pair quantitative metrics with a short narrative of what changed. The regulator will look for a line from test finding to action to verified closure to improved capability. If that line is broken, the metric is not evidence of continual improvement.