Eleven Hours Against a Four-Hour Promise

The conclusion is wrong, and it is the most important finding in this scenario - more important than any of the three technical causes.

The test measured one thing that passed: the team eventually completed failover. It measured against the stated objectives and failed both:

• RTO 4 hours, achieved 11 hours - a 7-hour gap, roughly 630,000 transactions at peak rates, plus a backlog and a counterparty exposure during the gap.
• RPO 15 minutes, replication lag 47 minutes - the recovery point was already breached before failover even began.

Calling this 'successful with observations' conflates completing the exercise with meeting the objective. That distinction is the whole point of testing. A test whose objective is missed is a failed test with useful diagnostic output, and the board needs to receive it that way, because the board's decision - accept the risk, fund remediation, or change the stated RTO - depends entirely on which framing it is given.

Two further problems:

• No retest was scheduled. A DR test produces findings; the retest is what converts findings into assurance. Without one, the organisation has documented that its recovery capability is inadequate and then done nothing to verify improvement.
• Board reporting was sanitised. Whether by intent or by optimism, the effect is the same: those charged with governance were told a capability exists that the evidence shows does not.

The auditor's recommendation should include the language used in reporting, not only the technical fixes.

Order by whether the failure is silent, continuous, and irreversible - not by which fix is easiest or was discovered loudest.

1. Replication saturation (first). The 47-minute lag is the only defect that causes loss without anyone noticing. A saturated link means the RPO breach exists every night, not just on test day, and in a real outage the lost transactions may be unrecoverable. Fix by rescheduling or throttling the batch job, bandwidth/QoS separation of replication traffic, and alerting on lag thresholds. Cheap compared with the other two and removes a persistent exposure.
2. Runbook and configuration management (second). 2.5 hours lost to a stale DNS reference is pure waste - no information was at risk, the team simply followed a document nobody maintained. Fix with version-controlled runbooks, a review cycle, and a requirement that the runbook be exercised, not just written. This is high-certainty recovery of time.
3. On-call availability and third-party escalation (third). 3 hours lost waiting for a single named individual is a single-point-of-failure in the people layer. Fix with a rotation, tested contact chains, contractual response times for the contractor, and cross-training. Slower to fix because it is organisational, but it removed more time than the DNS error in this test.
4. Retest with pre-agreed pass criteria and independent timing (last, but scheduled now). The retest validates 1–3 and is the only thing that produces actual assurance.

Why not start with the runbook because it is easiest? Because it recovers 2.5 hours of a 7-hour gap, while the replication fix closes an ongoing data-loss exposure that exists whether or not a failover ever occurs.

It is not primarily a DR finding at all - it is a production data-loss exposure that the DR test happened to expose.

Key implications:

• The RPO is breached during normal operations, not only during disasters. If replication lags 47 minutes nightly because of the batch job, then a production outage at 03:00 loses up to 47 minutes of transactions. The DR plan assumes 15 minutes. Nobody is reconciling the plan against observable replication behaviour.
• The lag was discovered by accident. A test caught it; the monitoring evidently did not, or it would have been a known issue. That absence of lag alerting is a separate control gap and arguably the root cause - the batch job has presumably been saturating the link for some time.
• The 47 minutes may understate the real figure. It reflects test-day conditions. If the batch job runs longer or heavier on month-end, the lag at those times is likely worse, and month-end is a higher-value period.
• RPO must be defined per data type and verified, not asserted. Ask how the 15-minute RPO was derived - business impact analysis, or a round number? A business that loses 47 minutes of payment instructions has a customer and regulatory problem, and the number in the BCP should reflect a business decision, not an infrastructure assumption.

Recommendation: instrument replication lag with alerting tied to the RPO, review lag during peak and month-end batch windows, and have the business confirm whether 15 minutes is genuinely required - if it is, the current architecture does not deliver it and that should be stated plainly to the board.