DR Test Misses RTO by Seven Hours: Severity, Priorities, and the 'It Has Never Happened' Argument

Adequacy. The capability is not adequate, and the test result is the least of the reasons. Four independent failures emerged, and their combination is what matters.

• The RTO was missed by more than seven hours, a 280% overrun, against a tier-1 system supporting clinical scheduling.
• The RPO of 15 minutes was missed by a factor of over 100, because the most recent usable full backup was 26 hours old. Data loss of that scale in a scheduling system means appointments, referrals, and staffing plans must be reconstructed manually.
• Two of five tapes failed checksum validation. This is a media and verification control failure, not bad luck; it means restore success was never verified before the test.
• The runbook referenced a decommissioned database version, showing the plan has not been maintained in step with production change.

Rating. High. The rating rests on the combination of a missed tier-1 objective, an unverified backup chain, and a recovery site sharing the primary's flood exposure. Any one of these alone might be moderate; together they mean recovery has never been demonstrated and may not be achievable at all.

Reporting. Report factually: objectives, measured results, the specific technical causes, and the last successful full test date. Note the two cancelled tests and the reason given. Avoid editorialising about management's attitude; the cancellation record speaks for itself. Attribute the finding to the process, and require a written response with dates and owners, plus a re-test date that the audit committee sees. Consider whether the finding rises to a reportable condition for the board risk committee given clinical impact.

Ordering should follow dependency and evidence, not ease. Six steps.

1. Fix backup verification first. Until restores are proven, nothing else is measurable; a plan that assumes restorable data is fiction. Implement automated restore verification with checksum validation and alerting on failure, and report a success rate weekly. This is cheap and fast, which makes it a good first visible win.

2. Re-baseline the RTO and RPO against clinical need, with the business owner signing. The current 4-hour figure may be inherited rather than agreed. If clinical leadership genuinely requires 4 hours, the architecture must change; if 12 hours is tolerable, the objective should say so. Either way, an unsigned objective cannot be tested.

3. Redesign the backup and replication approach to support the agreed RPO. Tape-based restore with a 26-hour gap cannot meet a 15-minute objective. This is the expensive step, so it must follow step 2 rather than lead it.

4. Address the site risk. A secondary site in the same river basin shares a single flood event with the primary. Relocation is slow and costly, so document interim risk acceptance while a regional separation plan is funded.

5. Make the runbook and contact list living artefacts. Assign ownership, require review on every major change, and test the call tree during the next exercise. These cost almost nothing and failed completely.

6. Institute a testing regime with defined scenarios, including one non-data-centre scenario such as ransomware, and a rule that change freezes cannot cancel a DR test without executive risk acceptance.

Why order matters. Steps 5 and 6 are trivial to complete and will create the appearance of progress while the system remains unrestorable. Doing the cheap items first and declaring improvement would be the predictable failure mode.

The argument confuses absence of evidence with evidence of absence, and it should be rebutted with the specifics of this test rather than with general statements about risk.

The historical sample is not informative. A short outage history over an unspecified period tells you the frequency of past events, not the capability to respond to the next one. Caldwell has never had to restore from a 26-hour-old backup during a flood, so there is no historical data on that scenario at all. The three-year gap since the last full test means most of the current staff have never executed a recovery.

The test itself is evidence of exposure. Recovery took 11 hours 20 minutes with a planned exercise, full staffing, and known scope. A real event occurs without notice, with degraded communications, competing priorities across systems, and staff who may be affected personally. Real recovery would almost certainly be slower. The measured capability, not the historical outage, is the relevant figure.

The exposure is not only outage duration. Failed tape validation means the restore set is unreliable now. The system is carrying latent risk every day, independent of whether an event occurs.

Reframe the discussion. The question is not whether an outage will occur but whether management is willing to accept a documented inability to meet a stated recovery objective for a clinical system. If so, that acceptance should be explicit, dated, and signed by the accountable executive, and recorded in the risk register. Most managers who are asked to sign such an acceptance change their position. Record the argument in the working papers along with the response; the auditor's job is to make the acceptance visible, not to win the debate.