The Metric That Rewarded the Wrong Behavior
A financial services firm reports continuity program health to its board through a dashboard. The headline metric is 'percentage of business units with an approved continuity plan,' which has risen from 64% to 98% over two years and is presented as proof of program maturity. Plan approval requires a unit head signature and an annual review date. A recent internal audit found that 40% of plans had not been meaningfully updated in over 18 months despite the review dates being current, several plans named employees who had left, and recovery procedures in three units referenced systems that had been decommissioned. No unit has ever failed an exercise, because exercises are optional and only four units volunteered last year. The board has asked why the program is 'essentially complete' yet the audit findings exist.
1. Critique the dashboard metric and explain how it produced the audit findings.
Read the worked answer
The metric measures documentation existence, not capability. 'Approved plan' is satisfied by a signature and a date - both administrative acts - so the program optimized for signatures and dates, which is exactly what it got. Rising from 64% to 98% measures the spread of paperwork, not the ability to recover.
The audit findings follow directly. Review dates are current because updating a date is trivial; content decays because nothing forces it to be tested. Departed employees and decommissioned systems persist because no mechanism connects plan content to current reality. And the absence of exercise failures is not a strength - with exercises optional and only four volunteers, the program has almost no capability evidence at all, so 'no failures' means 'no measurement.'
The deeper problem is that the metric is lagging, self-reported, and gameable, and it was presented as maturity. A board reading 98% reasonably concludes the work is done, which then undermines the case for continued investment.
2. Propose a replacement measurement framework and how to reset board expectations.
Read the worked answer
Replace existence metrics with a small balanced set covering capability, validation, and currency:
- Validation coverage: percentage of critical units that completed a exercise or test in the last 12 months, with results by capability, not pass/fail alone.
- Exercise findings: number and severity of gaps identified, plus closure rate and time-to-close - a healthy program finds gaps.
- Plan currency: percentage of plans whose named people, systems, and contact data were verified against authoritative sources within the last 90 days (automated where possible).
- Recovery evidence: results of actual restore or failover tests against RTOs for top-tier functions.
- Dependency coverage: percentage of critical third parties assessed.
Make exercises mandatory for critical units. Report trends and open risks, not a single completeness percentage.
Resetting board expectations: acknowledge plainly that the prior metric measured documentation and that 'essentially complete' was the wrong characterization. Present the audit findings as the baseline the new framework will track against, and show the board a forward path with targets. Boards accept corrected metrics when the correction is accompanied by a credible plan; they react badly to discovering the gap was known and unaddressed.