Certifying DR When the Drill Failed
Sunrise Bank's retail payment switch processes 8,000 transactions a minute at peak and routes IMPS, NEFT and UPI. Its primary data centre is in Mumbai; the DR site is in Hyderabad, 600 km away. In last week's annual drill the declared RTO was 2 hours; the switch was actually processing at 11 hours 20 minutes. Causes: the DNS zone was not replicated; 400 GB of switch configuration and certificate stores sat on a NAS outside replication; the switch licence is bound to the primary hardware serial; and 22 of 60 downstream interfaces were not redirected because the runbook owner left four months ago. The DR link is 100 Mbps against 1 Gbps at primary. Three critical vendors have no tested DR. The board must certify BCP readiness to the regulator in three weeks; the festive peak is ten weeks away; the next full drill is scheduled in nine months.
1. Can the board certify? Advise on what it can and cannot sign, and the wording you would recommend.
Read the worked answer
No unqualified certification. Certification is a statement about the adequacy of the plan and the evidence of testing; the drill result is evidence that the 2-hour RTO is not achievable. Signing an unqualified certificate would misrepresent the position to the regulator and expose directors personally.
What the board can sign: a qualified certification that (a) a BCP and DR plan exist and were tested, (b) the test demonstrated an actual recovery time of 11h20m against a stated RTO of 2h, (c) the gap is known, funded and tracked at board level, and (d) interim manual workarounds exist for the highest-impact services.
Wording to recommend: 'The board certifies that the BCP was tested on [date] and that recovery of the payment switch did not meet the stated RTO. Remediation is under way with [owner] accountable and re-test scheduled for [date]. The board has approved interim risk acceptance for the residual gap until re-test.'
Before signing, ask management: is the 2-hour RTO validated by a business impact analysis, or inherited from a template? If the analysis says 30 minutes for UPI, the gap is worse than reported and the certification must say so. Also confirm whether the drill outage is reportable as an incident - an 11-hour payment switch outage during business hours would be.
2. Prioritise remediation across the ten weeks to festive peak. What must be fixed, what can be mitigated, what can be accepted?
Read the worked answer
Must fix before the festive peak (weeks 1-6):
- Replicate the DNS zone and move the 400 GB of configuration and certificate stores into replicated storage. These are cheap, deterministic fixes with no dependency on the network upgrade.
- Resolve the licence: obtain a virtual or DR-site licence, or a vendor letter permitting rehosting. A serial-bound licence is a hard stop in a real failover.
- Rewrite the runbook and cross-train at least two people per critical interface; the departed owner is a single point of failure, not an excuse.
- Fix the top 15 downstream interfaces by revenue impact (IMPS, UPI, ATM, cards, reconciliation) and document manual workarounds for the rest.
Mitigate (weeks 1-10): upgrade or bond the DR link; at minimum measure actual replication throughput and set RPO expectations from it. Obtain vendor DR attestations for the three critical vendors and a contractual right to test.
Accept with sign-off: coverage of all 60 interfaces before the peak is not realistic. The board should accept the residual risk explicitly, with a named owner, and publish the manual fallback.
Run a switch-only mini-drill in week 6, not a full annual drill - test the fixes, not the paperwork. A second failed drill before the peak would be worse than no drill.
3. Using business impact analysis, challenge the 2-hour RTO and the replication design. What should the board be told?
Read the worked answer
A BIA-derived RTO would probably be much tighter than 2 hours for the switch.
- UPI/IMPS: near-real-time; a 30-minute outage at 8,000 transactions a minute is roughly 240,000 failed transactions, plus customer harm and reputational damage. An RTO of 30 minutes and near-zero RPO is defensible for the authorisation service.
- NEFT: batch-based; a 2-hour RTO may be acceptable if cut-off windows are respected.
- Reconciliation and reporting: 4-8 hours may be tolerable.
Challenge the number: if the BIA was never refreshed after UPI volumes grew, 2 hours may be an artefact. Ask for the financial impact per hour by service, the regulatory reporting deadlines, and the customer-harm threshold. Then set tiered RTOs and RPOs, and size DR investment against the cost of downtime - the 100 Mbps link may not support synchronous replication, which sets a physical floor on RPO that the board must see.
Also test the plan's assumptions: 600 km separation is good for site risk but introduces latency; if the DR site shares a seismic or flood zone with the primary, geographic separation needs re-examination. The board should be told that the current DR is a plan on paper, not a tested capability, and that the funding decision is between paying for recovery and accepting the loss.