Auto-Close Under Fire: Measuring AI Triage Precision Without Ground Truth

What makes this hard: analyst dispositions are a biased, noisy label source. Analysts never see what the model closed, so their labels cannot describe the auto-close population at all, and 15% "other" is effectively missing data. Treating dispositions as ground truth will flatter the model.

Sampling design

1. Stratify the auto-close population by alert type (EDR, cloud, identity, email) and by model confidence band (0.80-0.90, 0.90-0.97, >0.97). Precision is almost never uniform across bands, and a single global number hides the weakest one.
2. Draw a daily random audit sample per stratum and weight results back to stratum volume. Roughly 60 verdicts a day supports a monthly estimate with a confidence interval on the noisiest stratum; report Wilson intervals, not point estimates.
3. Blind the auditor to the model verdict, otherwise you measure agreement with the model rather than correctness.
4. Adjudicate disagreements between auditor and analyst disposition with a senior analyst. That adjudicated set becomes the first real ground-truth corpus and the regression suite for every future model update.

Escaped-defect tracking Sampling only finds misses inside the auto-closed bucket. Separately track incidents discovered through other channels (customer report, IR engagement, threat intel) whose precursor alert was auto-closed - that is the only direct signal of a false negative that left the SOC entirely.

Thresholds to propose Per-class precision at or above 99% for continued auto-close; no escaped defect attributable to auto-close in the trailing quarter; audit miss rate stable or falling. Any class that fails is demoted to human review until it passes two consecutive cycles.

Reasoning: the risk is not "the model is wrong sometimes" - every detector is. The risk is that a wrong verdict scales silently across 62% of volume with no human in the path. Controls must therefore act on the scope of autonomy, not on individual verdicts.

Control structure

1. Autonomy is granted per alert class, never globally. EDR malware-family closures and email phishing closures have very different error costs. A single switch means one weak class contaminates the trust case for the rest.
2. Confidence-band gating inside each class. Only the top band auto-closes; the middle band auto-closes with mandatory sampling; the low band routes to humans. This makes the autonomy boundary explicit and reviewable.
3. Shadow mode before promotion. A new model version runs alongside the incumbent for a defined period, with disagreements adjudicated, before it takes volume. Model updates are change-controlled like any detection content.
4. Separation of duties. The team that builds and tunes the model must not own the precision metric. The SOC risk owner signs off on continued autonomy using the audit results.
5. A tested kill switch that returns all classes to human triage within one shift, plus a documented re-enable procedure.
6. Verdict rationale retained per alert - the evidence and signals the model relied on - so an audit or post-incident review can reconstruct the decision months later.

Why this shape: it bounds damage per class, keeps the feedback loop independent of the model owner, and gives the CISO a lever that does not require dismantling the program.

First, treat rework as data, not disobedience. Re-opens are the SOC telling you the verdict is not auditable. Two very different causes produce identical symptoms, and you must separate them before acting.

Diagnose

• Sample 100 re-opens and score them: how many were genuinely malicious? A low hit rate means the re-open is anxiety-driven; a high hit rate means you have a real model defect and the analysts are right.
• Interview a handful of analysts. Typical drivers are: no visible rationale, a confidence score they cannot calibrate, and prior experience of being blamed for a miss.

Fix the information gap

• Surface the evidence the model used in the incident view - the signals, related alerts, and asset context - so the analyst can audit in seconds rather than rebuilding the case.
• Replace universal re-opening with a defined sample-based QA process: each analyst audits a rotating subset, with results published. This converts unstructured suspicion into a measurable control.
• Give analysts a first-class "disagree" action that captures a reason code and feeds the evaluation set. Disagreement becomes labeled data instead of duplicated work.

Close the loop visibly Publish a monthly note: how many disagreements were upheld, what changed, what the audit precision was. Analysts stop re-opening when they can see the feedback mechanism works.

Guardrail: never reduce rework by hiding the verdict or removing the ability to challenge. Keep an escalation path with a response SLA so skepticism has somewhere to go other than the queue.