Plan Document Nobody Can Execute at 3 A.M.
A financial services firm finalized a 210-page business continuity plan. It is organized by department, with each department's recovery steps in its own section. The plan references a separate 60-page IT disaster recovery runbook and a shared 'resources appendix' listing 300 contacts and 40 alternate sites. During a 2 a.m. outage of the core payments platform, the on-call incident manager spent 40 minutes locating the right section, found the payments steps spread across three department sections, and discovered the contact appendix listed a former employee as the primary escalation for the platform vendor. The recovery ultimately succeeded but took 3.5 hours longer than the plan's target. The plan had passed its annual review three months earlier, with each department head signing off on their own section.
1. What plan design and maintenance problems does this incident expose, and how should the plan be restructured?
Read the worked answer
Problems:
- Department-centric organization vs. incident-centric use. At 2 a.m., the user needs a single end-to-end recovery path for the payments platform, not fragments scattered across departments. The structure serves authors, not responders.
- Fragmentation across documents. Plan + DR runbook + resources appendix forces the responder to reconcile three sources under stress.
- Stale contact data. A former employee listed as primary vendor escalation means the contact data was never validated against a live source; department sign-off did not catch cross-cutting data.
- Sign-off theater. Each head approving only their own section left no one accountable for end-to-end coherence and currency.
Restructure:
- Organize around critical business services/processes, each with a single end-to-end recovery playbook spanning business and IT steps, with clear roles and decision points.
- Provide a front-page incident response guide: activation criteria, first 60 minutes, escalation, and pointers to service playbooks.
- Consolidate or tightly cross-reference the DR runbook and resources appendix; make the contact list a maintained, validated data source, not a static appendix.
- Assign an end-to-end plan owner accountable for coherence, plus periodic cross-functional validation.
2. How should the annual review and sign-off process change to prevent stale or incoherent content?
Read the worked answer
Move from document sign-off to validated capability and data currency.
- Validate critical data mechanically. Contact and escalation lists should be reconciled against HR and vendor records on a defined cadence (e.g., quarterly) and after any known change, with confirmation testing (call/acknowledge), not just list review.
- Test end-to-end, not section-by-section. Require at least one walkthrough or drill per critical service that exercises the full playbook across departments and IT, with findings logged.
- Assign an accountable plan owner for coherence, distinct from departmental content contributors who supply accuracy for their area.
- Tie sign-off to evidence. Sign-off should attest that the playbook was exercised and contact data validated within the period, not merely that the section was read.
- Track metrics: time-to-find-procedure in drills, contact-validation rate, action-item closure, and plan change log. Report these to management.
- Trigger-based updates (not just annual): org changes, vendor changes, system changes, and post-incident lessons should force a review of affected playbooks.