The Exercise That Passed and the Incident That Didn't
A payments processor runs an annual tabletop exercise on its cyber incident plan. For three consecutive years the exercise has been rated 'successful' by the exercise lead: participants identify the right escalation contacts, the crisis communication templates are located, and the decision log is completed. In November the firm suffers a real ransomware event. The response reveals that the on-call rota had not been updated for eight months, two of the four named crisis team members were on leave, and the decision to isolate the core payment switch took 5 hours because no one present had authority to authorise it. Post-incident review notes that every one of these factors was outside the scope of the annual exercise. The exercise lead argues the exercise was still valid because it tested the plan as written.
1. Assess the exercise programme's design against its purpose, and explain why three 'successful' exercises provided no assurance.
Read the worked answer
The exercises measured familiarity with artefacts, not capability under stress. Locating a template and filling a decision log tests whether documents exist and people can read them. It does not test whether the organisation can decide, staff, and act when conditions are adverse.
The three failures in the real event - stale rota, absent key personnel, absent delegated authority - are all structural and predictable, and all were excluded by design. That exclusion is the defect: an exercise scoped to confirm the plan is internally consistent will always pass, because the plan is always internally consistent. It tells you nothing about the plan's fit to reality.
The lead's defence ('we tested the plan as written') is precisely the problem. A plan as written assumes the rota is current and the named people are available. The exercise should have stress-tested those assumptions, not accepted them as inputs.
Assurance requires exercises that:
- inject unavailability of named roles, forcing deputies to act;
- require a real decision with a real authority question (e.g. 'authorise isolation of the payment switch - who signs, and how do you reach them at 02:00?');
- are audited against the rota and contact data as of the exercise date, not the plan's revision date.
2. Recommend how the exercise programme should change, and how success should be measured going forward.
Read the worked answer
Redesign around assumption-testing rather than document-walking. Concretely:
- Inject role absence by default. Every exercise removes at least two named crisis team members and requires the deputy structure to function. If a deputy does not exist, that is a finding.
- Test authority, not just notification. Include at least one decision that requires a named authorisation threshold, and time how long it takes to obtain. The 5-hour isolation delay becomes the benchmark to beat.
- Validate live data during the exercise: pull the on-call rota and contact list on the day and check it against reality. Stale data becomes a scored finding.
- Vary the scenario class across years so the programme is not over-fitted to one threat.
Measure success by findings generated and closed, not by participant satisfaction or completion of artefacts. A 'successful' exercise that produces zero corrective actions is a failed exercise. Track: number of material gaps identified, mean time to obtain a critical authorisation, percentage of injected absences successfully covered, and closure rate of actions within agreed dates. Report these to the same committee that saw the three prior 'successes', with an explicit note that the previous rating methodology was not fit for purpose.