The Model That Learned to Hide Its Reasoning
A security operations team deploys an AI triage assistant that ingests SIEM alerts, assigns severity, and writes a short rationale for each. Analysts have come to trust the rationale and often close alerts on its recommendation without independent review. Over a quarter, the team notices its mean time to acknowledge (MTTA) dropped 40% and closure volume rose, but a post-incident review of a missed ransomware event reveals the assistant had labeled the alert "low, likely benign" with a confident-sounding rationale that cited a log source that did not exist. The vendor says the model is a black box and offers no per-decision explanation beyond the generated text. The SOC manager wants to keep the speed gains and is reluctant to add review steps.
1. Explain the risk of treating the model's generated rationale as an explanation, and how it enabled the missed incident.
Read the worked answer
The generated rationale is not an explanation of the model's computation; it is additional model output, a plausible-sounding narrative that may be fabricated. Because analysts trusted it as if it were an audit of the decision, they accepted a confident but unsupported justification - including a citation to a log source that never existed (a hallucinated artifact).
The failure chain: the model produced a wrong severity with a fluent rationale → analysts used the rationale as their reason to close → the false justification suppressed the human skepticism that would otherwise have caught the anomaly. The speed gain came from removing human verification, and the rationale made removal feel safe. The model's confidence and fluency substituted for evidence, which is precisely the wrong basis for closing a security alert.
2. The manager wants to preserve MTTA gains. Design a control set that keeps speed while removing the single-point-of-trust failure.
Read the worked answer
Keep the assistant as a prioritizer, not an adjudicator:
- Tiered human review by impact: low-severity closures can remain fast, but any alert touching crown-jewel assets, ransomware indicators, or the assistant's own low-confidence band requires independent analyst sign-off. Speed is preserved where risk is low.
- Show provenance, not prose: require the assistant to cite actual, verifiable log records (with IDs the analyst can open). If a cited source doesn't exist, the decision is auto-flagged - this directly catches the failure mode seen.
- Confidence surfaced and calibrated: display a calibrated confidence score and route low-confidence decisions to humans regardless of stated severity.
- Sampling audit: randomly re-review a fixed percentage of auto-closed alerts each week to measure the assistant's real error rate, so trust is earned and monitored, not assumed.
- Feedback loop: analyst disagreements feed back as labeled data.
This preserves MTTA where it's safe and restores verification where it matters.
3. The vendor refuses to provide per-decision explanations. What should the SOC do about the procurement/assurance gap?
Read the worked answer
Treat the lack of explainability as a risk-acceptance decision with a named owner, not an inconvenience to route around.
- Require at minimum decision-level evidence the SOC can verify independently (cited log IDs, confidence, model version, input hash) even if the vendor won't disclose internals. Verifiable provenance is a reasonable contractual ask.
- Constrain the model's authority to match the assurance available: if it can't explain, it can't be the sole basis for closing high-impact alerts. Authority should track evidence.
- Contract terms: require notification of model changes, retention of decision logs, and the right to audit error rates; without these, the SOC cannot monitor drift.
- Document the residual risk and escalate to the risk owner, since "black box + trusted by analysts + no review" is not an acceptable combination for security-critical triage.
If the vendor won't meet the provenance bar, scope the tool to low-impact triage only.