Evaluation Suite That Passes While Users Complain
A customer-support multi-agent system has an offline eval suite of 120 cases that all pass at 96% task success. In production, the escalation rate to humans has climbed from 8% to 27% over six weeks, and CSAT on multi-issue tickets has dropped. The eval cases are single-issue, drawn from a 2023 snapshot, and were authored by the engineering team. Production logs show that 40% of real tickets contain two or more distinct issues, and that the triage agent frequently merges them into one thread, after which the resolution agent closes the ticket addressing only the first issue. There is no automated scoring of live traffic; the only production signal is the human escalation count.
1. Explain why the eval suite reports 96% while production degrades, and redesign the evaluation approach.
Read the worked answer
The suite measures a distribution the product no longer serves. Three specific gaps:
- Coverage gap: eval cases are single-issue; 40% of production is multi-issue. The exact failure mode driving escalation - issue merging and premature closure - is structurally untestable by the current suite.
- Staleness gap: a 2023 snapshot misses six weeks of drift in real tickets.
- Authorship gap: engineer-authored cases encode engineer assumptions, not real user language or edge cases.
Redesign:
- Sample the eval set from production traffic (with PII handling), stratified to match the real mix, so multi-issue tickets are ~40% of cases and each contains a check that every issue was addressed.
- Add a decomposition-specific metric: issue recall (fraction of distinct issues resolved), not just ticket-level success. A ticket that resolves 1 of 2 issues should fail.
- Continuous online evaluation on live traffic - automated scoring of resolution completeness and groundedness - instead of relying on escalation count as the only signal.
- Refresh on a cadence and track eval-set drift against production distribution.
The lesson: a passing eval is only meaningful if its cases resemble live traffic. Escalation rate is a lagging, low-resolution signal; it tells you something is wrong but not what.
2. The team wants to add an LLM-as-judge to score every production response. What safeguards and design choices are needed before trusting it?
Read the worked answer
An LLM judge is useful but is itself an unvalidated model; treat it as a measurement instrument that must be calibrated.
Safeguards:
- Validate against human labels first. Have humans score a sample (a few hundred responses), then measure judge–human agreement (e.g., Cohen's kappa or correlation). Do not deploy a judge whose agreement is unknown.
- Use a rubric, not a vibe. Explicit criteria - e.g., 'all distinct issues in the ticket are addressed,' 'no claim unsupported by retrieved context' - reduce variance and make disagreements diagnosable.
- Pin the judge model and prompt version. A judge upgrade silently changes your metric; version it and re-validate on upgrade.
- Guard against known biases: position/verbosity bias, self-preference for outputs from the same model family, and leniency drift. Randomize ordering where relevant and avoid using the same model as both generator and sole judge when feasible.
- Sample, don't necessarily judge 100%. Judging every response is costly; a statistically adequate sample with confidence intervals is often enough, with full coverage reserved for high-risk categories.
- Keep a human-in-the-loop audit on a small ongoing sample to catch judge drift.
The judge is a proxy metric. Its job is to move faster than human review, not to replace the ground truth humans provide.