Flood Warning, a Warm Standby, and Keys in the Wrong HSM

Pre-emptive failover is a risk trade, not a safety precaution in itself: it creates a guaranteed four-hour outage and EUR 180,000 cost to avoid a possible longer one. It also has a flaw worth naming immediately: a warm standby 40 kilometres downstream of the primary site in the same river system may be in the same flood envelope, so the failover target must be checked against the flood authority's modelled extent before anything else.

Decision logic:

• Use forecast triggers, not gut feel. Set gauge-level and time-to-crest thresholds agreed in advance with the flood authority, and define who may call the failover. Publish the matrix now, while the weekend is still ahead.
• Buy time cheaply first. Move backup copies and the HSM key material offsite, put the DR team on standby, pre-stage vendor engineers, and prepare the elective-surgery suspension decision separately from the failover decision. Many floods are forecast but do not reach the site.
• Compare real costs. EUR 180,000 against the cost of an unplanned outage at peak occupancy, plus the clinical risk of an improvised evacuation of 320 beds, usually favours pre-emptive action once the trigger is crossed, but only if the standby is genuinely outside the flood zone and has been proven to work.
• Record the decision and its basis, because either choice will be reviewed afterwards.

The 2022 test did not fail on infrastructure speed; it failed on a dependency nobody had mapped. Nineteen hours against an 8-hour claim means the plan is aspirational and the RTO is currently untrue, which is itself a governance finding.

Fixes:

• Escrow and replicate key material. The interface engine's keys must exist in the standby HSM or a split-knowledge escrow with a documented recovery ceremony. Backing up configuration without keys is backing up an encrypted brick.
• Map dependencies end to end. Interfaces, directory services, print, barcode medication administration, identity providers, DNS and the dispensing cabinets all gate clinical use even when the EHR is up. An RTO belongs to the clinical service, not the application.
• Automate and rehearse the runbook, and rehearse at least twice a year with the clinical teams who must work during the outage, not only with IT.
• Test against a defined pass criterion: EHR available and a real patient's medication ordered, dispensed and documented at the standby site within 8 hours. Anything less is a technology test, not a recovery test.
• Report the measured RTO to the board each quarter, with the gap to target and the remediation owner, so the claim is never again left untested for three years.

Set recovery objectives from clinical consequence, not from application criticality labels. Each service should be analysed by asking what harm accrues per hour of unavailability.

• Dispensing cabinets: a 6-hour battery is a hard constraint. Because medication release depends on network connectivity, the effective RTO is under 6 hours or the unit needs a documented override procedure with manual counts and pharmacist verification. Fund the override training and the paper stock, not just a bigger battery.
• Dialysis unit: 48 hours of paper flow sheets is the funded fallback, so a 48-hour RTO is coherent only if dialysis continues safely on paper. That requires printed schedules, a way to check allergies and current medications, and a documented reconciliation process. If prescriptions cannot be verified offline, 48 hours is not defensible and the RTO must shrink.
• PACS: a 24-hour RTO is often acceptable if prior images can be retrieved from the regional archive or the imaging vendor, and if reporting can continue on plain film or unenhanced review. Verify that fallback exists rather than assuming it.
• RPO follows the same logic: minutes for medication administration records, hours for imaging, because a repeat image is recoverable but a lost administration record is not.

Fund the fallback duration you actually claim. A paper process that has never been exercised is not a fallback, it is a plan to improvise.