Twenty-two months of incident data that proves nothing
A UK utility has run an incident process for 22 months. Data on 214 incidents shows 58% were first detected by third parties - customers, a supplier, a national CERT - and the median time from first observable evidence to detection is six days. Fourteen root causes recur: an unpatched VPN appliance, shared admin accounts, and unowned internet-facing assets. Of 96 corrective actions raised, 68 are past their due date. Mean time to resolve fell 22% year on year, but severity reclassification accounts for most of it: 31% of incidents were downgraded within 24 hours of opening. The annual tabletop produced 34 findings, of which five are closed, and the same scenario is reused every year. The board's risk committee now wants an incident dashboard and evidence the programme is improving. The CISO has one full-time equivalent for incident management governance and no tooling budget this year.
1. Critique the current measures and propose a balanced set with precise definitions, including how you would counter severity gaming.
Read the worked answer
What the current numbers actually measure. MTTR down 22% is not improvement if 31% of incidents were downgraded within 24 hours - the clock and the severity definition are both being managed. Third-party detection at 58% is a serious signal but, as a single figure, it does not say which control failed. Fourteen recurring root causes is the most useful number you hold, and it is not on the dashboard.
Balanced set, with definitions:
- Detection: median and 90th-percentile time from first observable evidence to declaration, split by source (internal tooling, employee, third party). This replaces the vague six days.
- Declaration discipline: proportion of incidents declared within target once a trigger is met; proportion reclassified after 24 hours, with reasons, reviewed rather than banned.
- Containment: declaration-to-containment time for priority 1 and 2, measured against a frozen severity definition.
- Recurrence: incidents sharing a root-cause category with a prior incident in the last 12 months. Target: falling.
- Corrective action health: percentage closed by due date, median age of overdue items, and percentage verified effective 90 days after closure.
- Exercises: findings closed within 90 days, and proportion of playbooks exercised in the last 12 months.
Anti-gaming controls. Freeze severity definitions for the reporting year, independently audit 10% of classifications, and treat reclassification as a quality metric rather than a failure. Publish the definitions next to the numbers so the committee can see what is and is not counted.
2. Design a lessons-learned process that actually closes corrective actions, given one FTE and 68 overdue items.
Read the worked answer
The problem is not the review meeting. It is that 68 of 96 actions have no owner with authority and no consequence for slipping. One FTE cannot chase 68 items, so change the mechanism rather than working harder.
Design:
- Tier the reviews. Priority-1 incidents and any recurrence get a facilitated review within 10 working days; everything else gets a lightweight written review by the incident manager, reviewed asynchronously. That is how one FTE covers 214 incidents.
- Cap and force-rank actions. No review may produce more than three actions; more than that means the review has not decided what matters. Each action gets one named person as owner, a due date, and defined evidence of completion.
- Split actions by type: fix (technical), control (process), and assurance (a test that the fix holds). Most programmes stop at 'fix' and never test it.
- Put the overdue list before the risk committee monthly, by owner. Escalation must be visible and social, not a reminder email. The FTE's role is curation and escalation, not delivery.
- Close the loop with reporters and affected teams - a short note on what changed. That is what sustains reporting.
- Retire stale actions explicitly, with a reason. An ageing backlog of actions nobody believes in is worse than a shorter honest list.
Test within two quarters: the overdue count should fall and recurrence should fall. If actions close but recurrence does not, the actions are the wrong ones.
3. How would you demonstrate improvement to the board within 12 months without new tooling? What evidence and baselines would you use?
Read the worked answer
Re-baseline first, or you will prove nothing. Recompute the six measures over the previous 12 months using the new definitions and state those definitions in the first board pack. A definition change mid-year must be disclosed, not hidden.
Credible evidence without new tools:
- A before/after on one narrow control. For the unpatched VPN appliance: show incident dates, the patch date, the compensating control, and the absence of further incidents from that cause. One closed root cause with evidence beats a dashboard of indices.
- Replay all 14 recurring causes against current controls: fixed, accepted risk, or not yet addressed, each with a date. That table persuades more than any maturity score.
- A different exercise each year, tracked to closure. Reusing the same tabletop yields the same 34 findings and teaches nothing. Run one unfamiliar scenario - supplier compromise or insider - and report finding closure, not finding count.
- Detection-source trend. If third-party detection falls from 58% to 40% while internal detection rises, that is measurable and attributable - provided the definitions do not move.
- Time-to-declare trend, because it is hard to game and is where the process most often fails.
Avoid a single composite maturity score. The committee cannot act on it, and it hides precisely the recurrence you are trying to fix. Give them three questions to answer: are we detecting sooner, are we repeating ourselves, and are our corrective actions getting done.