The Downtime That Exposed a Backup Problem
A community hospital's electronic record system goes down for six hours after a failed software update. During downtime, staff use paper forms. On restoration, the vendor's recovery process restores data to the state at 02:00, but the outage began at 09:00. Records created between 02:00 and 09:00 - including 14 medication orders and 3 discharge summaries - are found to be missing from the system after recovery. The paper forms captured most of the downtime activity, but some orders were given verbally and never written down. The hospital's business continuity plan says 'paper forms will be used during downtime' but does not specify reconciliation, and no one has been assigned to verify data integrity after restoration. The IT team wants to close the incident.
1. What went wrong, and what data-integrity risks remain after systems are restored?
Read the worked answer
What went wrong:
- Recovery point objective (RPO) mismatch. Restoring to 02:00 lost seven hours of data. The continuity plan apparently never defined an acceptable RPO, so no one noticed the gap until records were found missing.
- Incomplete downtime capture. Verbal orders were never recorded, so the paper trail is already incomplete before reconciliation begins.
- No reconciliation process. The plan covers how to work during downtime but not how to reunite paper and electronic records afterwards.
- No assigned accountability. Without an owner, verification does not happen.
Residual risks after restoration:
- Missing orders (14 medication orders) - patients may miss doses or receive duplicates if paper and electronic orders both exist.
- Missing discharge summaries - downstream clinicians and the patient may lack critical information.
- Duplicate or conflicting entries when paper records are later entered.
- Loss of audit trail for the 02:00–09:00 window, which matters for any safety review or legal claim.
- False confidence: the system 'looks' complete, so staff may not realise records are absent.
Closing the incident without reconciliation would leave these risks live.
2. Design the post-incident reconciliation and the changes to prevent recurrence.
Read the worked answer
Immediate reconciliation:
- Assign an owner (e.g. clinical informatics lead) accountable for closure.
- Reconstruct the timeline: list every patient with activity between 02:00 and 09:00 from paper forms, MARs, pharmacy logs, and staff interviews.
- Reconcile field by field: compare paper against the restored system; enter missing orders and summaries with a clear note that they were documented retrospectively, preserving the original time where known.
- Prioritise by risk: medication orders and discharge summaries first; verify each against the administering nurse and pharmacy.
- Verify with a second reviewer and sign off; document the reconciliation in the record.
- Notify affected clinicians and, where information was lost, the patients as required by policy.
Preventive changes:
- Define and test the RPO/RTO with the vendor; require evidence that backups are restorable to the correct point.
- Rewrite the continuity plan to include reconciliation steps, named roles, and downtime documentation standards (including a rule that verbal orders must be written immediately).
- Run downtime drills that include a simulated reconciliation, not just paper use.
- Add an automated post-restoration check comparing expected activity against restored data.
- Report the incident through the quality/safety and information-governance channels, with lessons learned.