Classifying a Failed Failover Test at a Credit Union
Fintrust Credit Union (core banking, 240,000 members, $3.1B assets) runs an annual full failover to a warm site 40 km from its primary data center. In this year's test, 78 of 86 Tier-1 services started. Online and mobile banking failed because Active Directory replication lagged by six hours between sites, so domain controllers at the warm site rejected authentications. The public DNS record has a 24-hour TTL, which was not lowered beforehand. The standby database was four days stale because a log-shipping job had been failing silently since the last change window; nobody noticed because alerts went to a mailbox of an engineer who had left. The exercise, planned to end at 14:00, concluded at 21:00 after the team fell back to the primary site. The DR manager wants to record the result as "Pass - with observations" to keep the board and the regulator satisfied. The core banking vendor's support contract requires an annual demonstrated failover.
1. Is "Pass - with observations" a defensible classification? What should the formal test report actually contain?
Read the worked answer
No - this is a fail with partial success, and the label matters
A pass, even a qualified one, asserts that the recovery capability exists. Here three independent controls failed simultaneously: identity replication (a six-hour lag that prevents authentication for every user), DNS (a 24-hour TTL that keeps production traffic pointed at the dead site), and data replication (a four-day stale database with a silently broken job). Any one of these alone would have broken the invocation; the fact that the app servers came up is not the measure of success. The test's purpose is to prove the service recovers within RTO, not that virtual machines boot.
What the report must contain
- Measured outcomes against targets, per service: planned RTO versus actual time to verified transaction, planned RPO versus actual data loss (here: four days, versus a stated target of minutes).
- Failure timeline with the specific technical cause of each failure and the detection gap - the log-shipping alert routing to a departed engineer's mailbox is a monitoring design defect, and the report should name it as such.
- Dependency findings - the exercise revealed that identity and DNS are upstream of everything and were not in the Tier-1 recovery sequence. That is a plan defect, not bad luck.
- Corrective actions with owners and dates, risk-rated, and the residual risk statement the board must accept until they close.
- Scope limitation note: the fallback to primary at 21:00 means end-to-end service restoration at the warm site was never demonstrated, which likely also leaves the vendor's contractual demonstration requirement unmet. Flag that to procurement rather than assuming it is satisfied.
A candid fail protects the institution; an inaccurate pass is a documented misrepresentation to a prudential regulator.
2. Prioritize the corrective actions and justify the ordering.
Read the worked answer
Order by what blocks everything else
1. Fix silent replication failure detection (highest). Restore monitoring on log shipping and AD replication with alerting to a monitored queue or on-call rotation, plus a second recipient. A stale standby database is the most dangerous finding because it is invisible until invocation and it silently destroys the RPO. Verify with replication health checks on a daily automated job, and add a recurring automated restore sample rather than trusting job status.
2. Make identity recovery explicit in the plan. Add domain controllers to the Tier-1 recovery sequence with a defined method: force replication and validate with replication summary tooling before application start; verify FSMO placement and site link costs. Applications cannot recover ahead of identity, so sequencing them later in the runbook is a design error.
3. Institute a DNS TTL change procedure in the invocation checklist. Lower TTLs to 60 seconds for all externally published records at least 48 hours before any planned test or forecast event, and pre-document the registrar/vendor change process and credentials. This is cheap and eliminates a full day of outage.
4. Then rebuild the test discipline. Test the full stack end to end within RTO before claiming recovery, and include known-failure injection (kill a domain controller, break log shipping mid-test) so the team rehearses detection, not just activation.
Why not start with the vendor or the board paper?
Contractual and governance items matter, but they do not stop the next outage. Ordering by "blocks the largest number of services" puts identity, replication, and DNS first; everything else is downstream.
3. The next scheduled full test is 11 months away. What should happen before then, and what type of exercise do you run?
Read the worked answer
Do not leave a known-broken recovery capability untested for a year
A failed test means the organization currently has no demonstrated recovery capability for core banking. Waiting 11 months is an unmanaged risk, not a schedule.
Interim sequence
Within 30 days - remediation verification (targeted test, 4–6 hours). Retest only the failed components: force AD replication and authenticate users at the warm site, lower TTLs and verify record propagation, validate standby database currency with an actual query against a known transaction. This is a component-level test, not a full failover, and it is cheap enough to repeat until clean.
Within 60–90 days - checkpoint failover of Tier-1 only, during a low-risk window. Full core banking failover, measured against RTO, with the business participating so that transaction verification is real. Schedule it outside month-end, payroll runs, and any regulatory reporting window; the temptation to test during a blackout period is exactly why failovers get deferred indefinitely.
Within 6 months - tabletop with the executive and crisis comms teams. The 14:00-to-21:00 overrun shows a decision-making problem as much as a technical one: who declares a disaster, who authorizes fallback, who talks to regulators and members. Discussion-based exercises are cheap and they exercise the part of the plan that no failover test touches.
Also fix the contractual clock
Confirm with the vendor whether the failed exercise satisfies the annual demonstration requirement. If it does not, obtain written confirmation of the remediation plan and a new demonstration date - do not let an unmet contractual condition surface during a support dispute.