A Warm Standby That Has Never Been Warm
You are the infrastructure lead for a three-hospital system. The EHR runs in a single primary data center; a warm standby 40 miles away replicates asynchronously with an observed 6- to 12-minute lag and has never been activated in production. During a fiber cut last year, clinicians reverted to paper for four hours, and 11 orders were never transcribed into the record afterward. The CIO has $1.2 million and wants clinical downtime risk reduced within 12 months. Options on the table: synchronous replication to the standby ($2.4M), a second fiber path from a different carrier with diverse routing ($300K), a read-only downtime viewer refreshing every 15 minutes ($150K plus $40K/year), and stronger paper downtime procedures with quarterly drills ($60K). The network team reports that the last three unplanned outages were all WAN or carrier failures, not data-center failures.
1. How would you frame the risk before recommending anything?
Read the worked answer
Define downtime risk clinically, then map it to failure modes before spending a dollar.
Enumerate the modes: WAN/carrier failure, data-center power or cooling loss, application or database corruption, ransomware, and planned maintenance. For each, estimate likelihood, likely duration, clinical impact, and - separately - how fast the organization would detect it.
Two facts from this environment dominate the analysis:
- The last three unplanned outages were WAN or carrier, not data-center events. Synchronous replication protects against site loss - the failure you have not experienced - and it faithfully replicates logical corruption and ransomware to the second site, which can turn an incident into a catastrophe.
- The 11 lost orders were a reconciliation failure, not an availability failure. The system came back in four hours; the paper-to-EHR reconciliation never happened. Buying availability without fixing reconciliation leaves the same patient harm in place.
Force the clinical conversation this data should trigger: what recovery time and recovery point are acceptable for order entry, results review, and medication administration? Ask nursing and pharmacy, not only IT. And require a physical route survey - a second carrier means nothing if both fibers enter through the same conduit.
2. How would you allocate the $1.2 million, and what would you explicitly decline to buy?
Read the worked answer
Recommended allocation:
- $300K - second-carrier fiber path, contingent on documented physical route diversity. Highest value per dollar against the dominant observed failure mode.
- $150K plus $40K/year operating - read-only downtime viewer. Cheapest way to let clinicians see recent results, medications, and allergies during an outage. Accept the 15-minute refresh as a known limitation and write it into the downtime procedure.
- $60K - reengineered downtime procedures and quarterly drills, explicitly including a paper-to-EHR reconciliation workflow with a named owner and a post-downtime chart completion audit. This directly addresses the 11 lost orders.
- Roughly $690K held in reserve - fund measured improvement in replication (lag monitoring, replication health alerting, a tested failover) and use two quarters of drill and outage data to justify the next increment.
What I would decline: synchronous replication at $2.4M. It is double the budget, addresses the least-evidenced failure mode, replicates corruption, and would consume the funds that fix network diversity and reconciliation. If the CIO wants it eventually, it should be argued with data from the drills, not bought first.
3. What evidence would you require before allowing anyone to call the standby site operational?
Read the worked answer
- A scheduled failover exercise with clinical staff present - not an infrastructure-only test - followed by a full failback.
- Measured RTO and RPO reported as actuals against the clinically agreed targets, not as vendor specifications.
- Interface verification at the standby site: laboratory, pharmacy, radiology, and the interface engine all sending and receiving.
- Identity and credential validation, including temporary access for staff who do not normally work at that site.
- Device-dependent workflow checks: label printing, scanners, pharmacy automation, workstation mapping, and the downtime viewer itself.
- A communication plan exercised end to end, including how clinicians learn they are on the standby site and who declares the event.
- Contract and licensing confirmation that running production workloads at the second site is permitted and vendor-supported.
- A retest cadence - at least annually and after any major change - with results reported to a governance body rather than filed.
Until those exist, the standby is a backup, not a recovery capability. The distinction matters most in the sentence a CIO says to a board.