A Remediation Metric the Board Should Not See
Orion Media's security dashboard reports a mean time to remediate of 41 days across 120 projects. The CISO's board report repeats 41 days; engineering says the real number is closer to 90. Investigation shows four problems. Vulnerabilities are auto-resolved when a scanner stops detecting them, including when a dependency is removed and when a project simply stops being scanned. About 38 percent of findings are dismissed with the reason false positive but no comment. Twelve projects use severity overrides to lower Critical findings to High. Only 55 of 120 projects run container scanning, and 22 projects have not run any scan in 90 days. The vulnerability report includes findings from feature branches that never merged. The board has asked for one defensible number and a plan to improve it.
1. Why is the 41-day figure not defensible? Identify each mechanism that distorts it.
Read the worked answer
The figure is a pipeline artifact, not a measurement of remediation. Auto-resolution is the largest distortion: when a scanner stops detecting a finding because the dependency was removed, the finding may genuinely be fixed, but when the project stopped scanning, the finding disappears without being fixed. Both events look like a fast resolution and pull the mean down.
Scope is the second distortion. Including feature branch findings measures work that may never ship, and it inflates the denominator with findings whose lifecycle is unrelated to production risk. The metric should be tied to the default branch and to deployed images, not to every branch.
Dismissal is the third. A 38 percent dismissal rate with no comment means an unknown portion of findings were closed without evidence. Those dismissals count as resolved in many reports, which makes the organization look faster while hiding risk.
Severity overrides are the fourth. Lowering Critical to High changes both the SLA and the reported mix, so the board sees fewer Critical findings without any remediation. Finally, coverage is unequal: 55 of 120 projects run container scanning and 22 have not scanned in 90 days, so the population is not comparable across projects. A mean over a changing, incomplete population is not a defensible metric.
2. Define one defensible remediation metric and the data hygiene rules required to compute it.
Read the worked answer
Use a cohort metric: median days from first detection to remediation for confirmed Critical and High vulnerabilities on the default branch in production projects, with the cohort defined by detection date and reported alongside scanner coverage. Median rather than mean, because a few very old findings distort the mean. Report the open backlog age and the percentage past SLA as companions, because a median can look healthy while a long tail sits unresolved.
Data hygiene rules: restrict the population to default branch and deployed images. Require a comment and an approval from someone other than the author for every dismissal, and report dismissed findings separately rather than folding them into resolved. Restrict severity overrides to the application security team with a written justification, and audit them monthly. Track auto-resolved findings separately and verify a sample against the scanner and repository state; if a project stopped scanning, reopen the findings that were auto-resolved by the coverage gap. Publish coverage as a percentage of projects scanned in the period, and do not report the metric for a month in which coverage fell below an agreed threshold without a caveat.
Set SLAs by severity, for example 7 days for Critical and 30 days for High, and report the breach rate. The board number becomes the median plus the breach rate plus coverage, which cannot be improved by simply scanning less.
3. What would you change in the next 90 days to improve both the metric and the underlying risk, given limited engineering time?
Read the worked answer
First, fix coverage before optimizing speed. Apply a scan execution policy that runs container scanning and dependency scanning on all 120 projects on a schedule, and make scanner coverage a monthly reported KPI. Without coverage, any improvement in the median is suspect. The 22 projects with no scans in 90 days are the first target because they are the biggest blind spot.
Second, enforce triage quality. Require a comment for dismissals and route dismissals of Critical findings to the application security team. Review the existing 38 percent dismissal backlog in the highest-risk projects and reopen anything dismissed without evidence. Remove blanket severity overrides and re-evaluate the 12 projects that used them.
Third, put a gate where it changes behavior. Add merge request approval policies for new Critical findings in the production projects, so the cheapest fix happens before merge rather than months later. This reduces the inflow that drives the median.
Fourth, assign ownership. Map findings to teams through CODEOWNERS and project ownership, and give each team a monthly SLA breach report. Do not add a new dashboard until the existing data is trustworthy. Recompute the cohort metric monthly, publish it with coverage and breach rate, and explain the first two months as a baseline rather than a trend.