Thirty-six hours before the river reaches the data center
You are the continuity lead for a three-hospital regional health network. The electronic health record is hosted in a data center sited in a mapped 100-year floodplain. The continuity plan states a four-hour recovery time objective for the EHR, supported by a warm site 90 miles away under a contract last exercised in a full failover two years ago; that test restored service at six hours ten minutes with 40 minutes of record loss. VPN licensing supports 200 concurrent remote users against 900 staff who would need remote access in a disruption. The river forecast now puts the data center in the inundation zone in 36 hours with moderate confidence; the peak forecast carries a 40% chance of exceeding the levee.
1. Do you invoke now or wait for better forecast confidence? Reason it out explicitly.
Read the worked answer
The four-hour RTO is not the number to plan against. Your only full test delivered 6h10m. Plan against demonstrated performance, not the stated objective, and add decision latency - the time between noticing a problem and someone with authority saying go. Call effective lead time eight hours, minimum, and treat it as the constraint.
With 36 hours, you are not choosing between invoking and not invoking. You are choosing whether to spend the next 12 hours on cheap reversible preparation. Do that now: verify replication and backup integrity, notify the vendor in writing that activation is likely and confirm who can authorize it, confirm the clinical downtime procedures are current, and set a formal decision point - say 1800 today, or the next forecast cycle, whichever comes first - with a named decision-maker.
Reasoning: pre-positioning costs a few thousand dollars and some staff time. A late failover costs clinical operations during a live evacuation, when the network is already stressed. But a full failover at moderate confidence is not free either: it introduces its own change-related risk to a live clinical system, and doing it twice destroys credibility.
State the trigger in advance: if the forecast places the data center in the inundation zone with high confidence, or if the levee exceedance probability rises above a defined threshold, you activate. Writing the trigger down before the pressure arrives is what keeps the decision from becoming a negotiation.
2. Given 36 hours, what do you fix and what do you accept?
Read the worked answer
Fix, in priority order:
- Backup and replication integrity. Verify the most recent restorable copy and its location. Forty minutes of record loss in the last test is a data integrity problem, not a speed problem. Confirm the offsite copy is not in the same floodplain.
- VPN capacity. 200 licenses against 900 staff means 700 people cannot work remotely. Find out whether the vendor offers burst or temporary licensing on hours' notice - many do - and whether priority roles can be defined so the 200 slots go to the people who need them. Pre-assign the slots now.
- Clinical downtime procedures. Confirm every unit has current paper forms and that staff on shift tonight know the procedure. This is done in hours and it is the mitigation that matters most if the failover is slow.
- Vendor activation authority. Written confirmation of who calls, who answers, and what the contractual response obligation is.
Accept, and say so: the gap between 4 hours and 6h10m will not close in 36 hours. Do not attempt contract renegotiation, new infrastructure, or a licensing overhaul mid-event - those consume the staff you need for items 1 through 4. Accept a documented residual risk of approximately two hours of degraded EHR performance plus the data-loss exposure, and escalate that acceptance to the incident commander in writing so it is a decision, not an omission.
3. After the event, how do you turn the failover gap into an approved corrective action?
Read the worked answer
Do it whether or not the river wins. A forecast that did not materialize is still a near-miss and the cheapest evidence you will ever get. If the levee holds, you have a documented test of your decision process and a live stress test of your vendor contacts - write it up before the memory softens.
Quantify in the language the executive team uses. A 2h10m gap is not a number; it is [X] admissions per hour with no order entry, [Y] medication orders on paper, [Z] diversions if the gap lands on a high-census day. Combine with the 40-minute data loss as a clinical-risk and record-integrity exposure. Attach a regulatory consequence if one applies - accreditation standards and health record retention requirements carry more weight with a board than an operational preference.
Propose a tiered fix, not a number. Tier 1: burst VPN licensing and pre-assigned priority roles (low cost, resolves a third of the exposure). Tier 2: a second full failover test within twelve months with a written acceptance criterion (staff time only). Tier 3: relocate or re-architect the hosting arrangement (capital, multi-year). Tiers 1 and 2 are approvable now; tier 3 needs a business case.
Close the loop formally: a corrective action with an owner, a date, and an executive sponsor. If leadership declines, record it as an accepted risk with the accepting executive named. An unrecorded risk is a future finding against you; a recorded one is a governance decision.