LLM Failure Summary in a CI Dashboard
SaaSly runs 600 UI and API tests nightly. The CI dashboard sends failed logs to an LLM that produces a one-paragraph summary and a severity label for the daily triage call. Last sprint the summary grouped 14 unrelated failures as “payment gateway timeout,” omitted a real authorization defect because the relevant log lines were truncated at 8 KB, and labeled six DNS failures as P1 product defects. It reported “95% pass rate” although 3 tests were skipped and 2 were quarantined as flaky. Engineering managers now use the summary in release go/no-go decisions without opening the raw logs. The test team has two hours each morning for triage. The SRE team owns infrastructure alerts, and the security team requires immediate escalation of any access-control anomaly.
1. Assess the reliability and risks of the LLM-generated summary for triage and release decisions.
Read the worked answer
The summary is a lossy compression of evidence, not a source of truth. Truncating logs at 8 KB can remove the exact stack frame or authorization message needed for diagnosis, producing false negatives. Grouping 14 unrelated failures under one label collapses distinct causes and misdirects triage. Severity labels are not grounded in product impact, so DNS failures become P1 product defects. The reported pass rate is also wrong because skipped and quarantined tests are excluded from the denominator without disclosure.
The main risk is not only a bad summary; it is managers making go/no-go decisions without opening raw evidence. A missed access-control defect is a security and compliance risk. The team should treat the LLM output as a draft narrative that must link to raw logs, deterministic metrics, and human triage. Security anomalies must bypass the summary and escalate directly.
2. Design validation, monitoring, and escalation controls for the AI summary.
Read the worked answer
Compute pass, fail, skip, and quarantine counts deterministically from raw results before any LLM call. Display the denominator and define pass rate over executed tests, with skipped and quarantined tests shown separately. Require every summary claim to link to raw log lines and test IDs; when evidence is missing, the model must output “insufficient evidence” rather than guess.
Audit a sample of 20 failures per day against human triage and track grouping precision, severity precision and recall, and false-negative rate. Compare summary output with the raw logs for truncation and missing sections. Route access-control and authentication anomalies directly to security, and infrastructure failures to SRE, bypassing the LLM. Give triagers a one-click correction that feeds a weekly review of mislabels, prompt changes, and log-retention fixes.
3. Recommend reporting metrics and a decision process that uses the LLM without letting it decide release readiness.
Read the worked answer
Report source-of-truth metrics: executed pass rate, skipped and quarantined counts, new vs known failures, flake rate, defect leakage, triage time, and misclassification rate from audits. Add MTTR for product defects and a security-escalation SLA. The LLM may draft the narrative, but release readiness must be decided by a human using the raw metrics and linked evidence.
Define an escalation rule: any security anomaly, any unexplained drop in executed tests, or any summary with low evidence coverage requires human review before the go/no-go call. Track whether corrections reduce repeated errors over time. This is continuous improvement: the feedback loop improves prompts, log capture, and metric definitions rather than merely making the summary sound better.