A Green Dashboard and a Fourteen-Month-Old VPN Appliance
Orion Payments reports to its board each month: patch compliance 99.4%, phishing click rate 3%, open critical vulnerabilities 0, training completion 97%. In March an attacker entered through an internet-facing VPN appliance that had been unpatched for 14 months and moved laterally using a local administrator account shared by the database team. The post-incident review established that patch compliance was calculated only over servers reporting to the scanner - 1,410 of 1,860 known assets - and that the VPN appliance sat in a DMZ excluded from scanning because 'the scan breaks it'. 'Open critical vulnerabilities' counted only CVSS 9.0+ findings from a single monthly scan. The phishing figure excluded users who reported the simulation. Training completion counted finishing a four-minute video. The board asks why the dashboard was green, and whether the metrics were wrong or the reporting was. The CISO wants a revised measurement set before the next board meeting and an answer on who should have caught this.
1. Diagnose why the dashboard produced false assurance. Separate coverage, definition and validity problems, and say which one would have caught this incident.
Read the worked answer
Coverage. The denominator was wrong. Compliance was measured over assets that reported to the scanner, not over assets that exist: 450 known assets were outside the calculation and the appliance was excluded by design. Any percentage computed over a self-selecting population will look good, and this is the single defect that would have caught the incident - the appliance was unpatched because it was never scanned, so a coverage metric (scanned and reporting / known assets) would have surfaced it immediately.
Definition. 'Open critical vulnerabilities' counted only CVSS 9.0+ from one monthly scan, so 8.0–8.9 findings were invisible and appliance-class vulnerabilities that no agent can see were never counted. The phishing metric excluded reporters, which removes the best behavioural signal and rewards the wrong thing. Training completion measured a video, not competence.
Validity. Even with correct coverage and definitions, these are activity measures. None of them measures whether the control is effective: a scanned-and-patched server tells you nothing about whether the patch was applied to the right component, or whether compensating controls exist where patching is impossible.
So: were the metrics wrong or the reporting? Both. The metrics measured a convenient subset and presented it as the whole; the reporting never disclosed the population, the exclusions or the definitions. Green was a threshold artefact, not an assessment. The missed signal was coverage and exception ageing, not a mystery.
2. Redesign the measurement set. Specify each metric's numerator, denominator, source of truth and the decision it drives, and distinguish leading from lagging indicators.
Read the worked answer
Every metric needs a definition, a population, an owner and a decision. A dashboard nobody acts on is decoration.
Leading indicators (change before harm):
- Scan coverage = assets reporting and scanned in the last 7 days / assets in the configuration management database, reported by tier. Source: scanner plus CMDB. Decision: fund discovery and onboarding; block new assets that are not registered.
- Unmanaged asset count, trended weekly. Decision: hunt and register; escalate when flat or rising.
- Exception and risk-acceptance ageing = open exceptions past their review date / all open exceptions, split by internet-facing. Decision: renew, remediate or escalate to the risk owner.
- Time to remediate by exposure tier (median and 95th percentile), not a compliance percentage. Decision: reallocate remediation capacity; internet-facing and no-compensating-control findings get the shortest clock.
- Privileged account hygiene = privileged accounts per person, percentage vaulted with session logging and multi-factor. Decision: remove standing privilege.
- Phishing report rate and time-to-first-report, reported alongside click rate, never instead of it. Decision: tune detection and recognise reporters.
Lagging indicators (confirm outcomes): incidents by root cause; mean time to detect and contain; successful restore tests.
Presentation rules: always show the population and the exclusions next to the number; show trends, not single months; and require a stated decision for each metric. If a metric cannot change a decision, drop it from the board pack and keep it in operational reporting.
3. Explain how monitoring, technical compliance review and internal audit should interact here, and what corrective-action governance you would put in place. Who should have caught this?
Read the worked answer
Three different jobs, often conflated:
- Monitoring (8.16) is continuous and operational: scanner coverage, log correlation, alerting. It should have caught an unpatched, internet-facing appliance - coverage reporting would have flagged the asset as unscanned within a week.
- Technical compliance review (8.8) is periodic, independent checking that controls are implemented as documented: sample devices and confirm they are in the scanner, patched, and covered by an exception if not. This is the layer that would have caught the deliberate DMZ exclusion, because it questions the exclusion rather than inheriting it.
- Internal audit (9.2) provides independent assurance over the management system, including whether the metrics themselves are trustworthy: does the CMDB match reality, are definitions followed, is the board pack honest? Audit should sample the population rather than re-run the same report.
Who should have caught it: the asset and vulnerability owners first, for excluding a device without a compensating control or an approved exception; then technical compliance review, for never testing the exclusion; then management review, for accepting a dashboard without asking what the denominator was. Distributing blame without fixing the exclusion process guarantees a repeat.
Corrective-action governance (10.1, 10.2): every nonconformity gets a documented root cause (not 'human error'), a named owner, a due date, and a defined verification of effectiveness - not just evidence that an action was completed. Track corrective actions in one register reviewed monthly by the security leadership team and quarterly by management review, with overdue items escalated. For this incident: onboard the appliance to scanning with a documented compensating control, inventory all DMZ and appliance-class assets, redefine the metric with a disclosed population, and verify after 90 days that coverage is genuinely above target before reporting green again.