A Reassurance Programme That Reassures Nobody
Halcyon Bank's security operations centre has 11 analysts and ingests around 4,000 alerts a day. Its quarterly dashboard reports 92% patch compliance, a mean time to detect of 9.2 days for 'non-critical' incidents, and 61% coverage of MITRE ATT&CK techniques against its detection use cases. Last quarter the vulnerability scanner reported zero critical findings on the externally exposed payments portal, yet an external penetration test found three criticals on the same portal, including an authentication bypass. Separately, 6,400 of 19,000 endpoint agents have not checked in for more than 60 days, and the patch compliance figure excludes them. The last purple-team exercise was 26 months ago. The board risk committee has asked the CISO for a one-page statement on whether controls are 'operating effectively', and internal audit has scheduled a review of security monitoring next month.
1. Assess the assurance value of each reported metric. Which are misleading and why?
Read the worked answer
Each metric answers a different question than the one the board is asking, and two are actively misleading.
- 92% patch compliance - misleading. The denominator excludes 6,400 silent agents, roughly a third of the estate. The true figure could be under 70%, and the silent population is exactly the population most likely to be unpatched. A compliance percentage that omits unknown assets is a measurement of the known-good subset, not the estate.
- MTTD 9.2 days for non-critical - not informative. Aggregating across a 'non-critical' bucket hides the distribution and excludes the alerts that matter. A useful measure is time to detect and time to contain by severity, with the tail reported, not just the mean.
- 61% ATT&CK coverage - weak evidence. This counts mapped use cases, not validated detections. Mapping is an authoring exercise; coverage claims require adversary emulation to demonstrate that a technique is actually detected.
- Scanner: zero criticals vs pentest: three criticals - a direct contradiction. Either scanning is unauthenticated, plugins are stale, the portal was excluded, or the scanner cannot see the authentication logic the tester exploited. Any of these invalidates the scanner as assurance for that application.
- Purple team 26 months ago - stale. Two years is several architecture generations in a bank.
The honest summary: the dashboard measures activity and configuration intent, not effectiveness, and the one independent test conducted contradicted it.
2. Design a validation approach for the patch management control that internal audit would accept.
Read the worked answer
The control must be tested against the population it claims to cover, by a party that is not the team producing the metric, using evidence that cannot be generated by the same tool that generated the claim.
Steps:
- Reconcile the population. Compare the scanner's asset list, the EDR agent list, Active Directory computer objects and the CMDB. The 6,400 silent agents must appear as findings, not exclusions. Any externally reachable asset among them is an immediate escalation, not an audit line item.
- Fix measurement before testing it. Enable authenticated scanning, verify credential health, and confirm plugin currency. Re-scan a sample of hosts where the pentest found criticals - if the scanner still reports clean, the scanner is the finding.
- Sample by risk tier, not uniformly. Take all tier-one and internet-facing assets, then a stratified random sample across operating system and patch ring for the remainder. Sample size should be justified by the confidence the committee needs and the tolerable error rate - a 5% sample of tier-two servers with zero exceptions supports a different conclusion than the same sample with three.
- Test end to end. For a sample of patched hosts, trace from vendor bulletin to deployment record to verified installation on the host, using host-based evidence rather than console status.
- Test the exceptions process. Open risk acceptances, emergency patch records, and change tickets for out-of-band deployments.
- Re-test after remediation and report both the finding and the retest result.
Report exceptions with asset names, exposure and remediation dates so audit can sample the remediation itself.
3. Draft the substance of the one-page effectiveness statement for the board risk committee.
Read the worked answer
The statement must be short, quantified, and honest about uncertainty. It should not be a defence of the dashboard.
Suggested content:
- Conclusion first: 'We cannot currently assert that security controls are operating effectively across the full estate. We can assert effectiveness for approximately 66% of endpoints and for the payment portal as tested by external penetration testers, which found three critical issues that our own scanning missed.'
- Evidence for the conclusion: silent agent population, scanner-versus-pentest contradiction, age of the last adversary emulation, and detection coverage based on mapping rather than testing.
- What is working: alert triage capacity, the external test itself, and the fact that the contradiction was surfaced rather than buried.
- What is not measured: detection efficacy against real adversary behaviour, coverage of the 6,400 silent assets, and control performance in the OT or third-party estate if applicable.
- Remediation plan with dates: agent reconciliation complete within 30 days, authenticated scanning and plugin refresh within 45 days, pentest criticals remediated or formally risk-accepted within agreed SLAs, purple-team exercise scheduled within 90 days with results reported to this committee.
- Risk statement: residual risk is elevated and, for the unmeasured population, unknown; unknown should be recorded as high until measured.
- Decision requested: endorse the remediation plan, accept the interim risk statement as written, and require a follow-up report with measured results.
One page means one page - put the detail in an appendix and keep the conclusion and the decisions in front of the committee.