The Exercise That Was Designed to Pass
A payments company runs an annual tabletop exercise. For three consecutive years it has scored 'fully met' on all objectives, and the exercise report goes to the audit committee as evidence of readiness. A new resilience lead reviews the materials and finds: the scenario is a generic 'server outage' resolved in 90 minutes; the facilitator is the IT director whose own team owns the recovery; participants receive the scenario and the expected answers one week in advance; there is no inject beyond the opening; and the same 11 people attend every year. Separately, a real 4-hour payment gateway failure eight months ago was handled chaotically, with no one able to locate the current contact list. The next exercise is in ten weeks and the audit committee has already been told the program is 'mature'.
1. Explain why the exercise program is producing false assurance, and what the recent real incident reveals that the exercises did not.
Read the worked answer
The exercise is designed for confirmation, not discovery. Each design choice removes the possibility of failure: a short generic scenario never stresses decision-making under ambiguity; pre-briefing the expected answers tests memory, not capability; a single inject means no escalation or surprise; the facilitator owning the system under test creates a conflict of interest and suppresses challenge; and a fixed participant group builds a closed social loop where everyone knows the drill. A 100% pass rate across three years is not evidence of maturity - it is evidence the exercise cannot detect weakness.
The real incident is the true test result, and it failed on a basic capability: contact list currency. That is exactly the kind of finding a well-designed exercise surfaces, and its absence from three years of 'fully met' reports is the proof of false assurance. The gap between reported readiness and observed performance is the finding the audit committee needs.
Frame this as a measurement validity problem: the program measured the wrong thing and reported it as assurance. That framing protects the new lead from appearing to attack colleagues while making the seriousness clear.
2. Design the next exercise cycle to generate genuine evidence, and address the audit committee communication.
Read the worked answer
Redesign around realistic stress and independent facilitation. Use a plausible, escalating scenario tied to a real threat (e.g., a ransomware event affecting the payment gateway with degraded, not absent, telemetry). Deliver injects in real time with no advance script, including at least one that invalidates an early assumption and one requiring a decision with incomplete information. Move facilitation to someone without ownership of the systems under test, or bring in an independent facilitator. Rotate and expand participants, and include a communications and call-tree component that directly tests the failure seen in the real incident. Add a no-notice component or a component that tests actual restoration or failover rather than discussion. Capture observations against explicit capability objectives with evidence, and grade honestly, including 'not met'.
On the audit committee: do not wait ten weeks to correct the record. Send a brief now stating that the prior exercise methodology had limited ability to detect gaps, that a real incident exposed a contact-list failure, and that a redesigned program will report evidence-based results at the next cycle. Present the corrective action as strengthening assurance, and include the real incident as a case finding. Correcting an overstated maturity claim proactively is far less damaging than having the audit committee discover the gap itself.