Calibrating an LLM Judge for a Support Copilot
A support copilot handles 60,000 conversations a day. Quality is measured by an LLM judge on 100 sampled conversations per day, scoring groundedness and helpfulness. A human review of 200 conversations found the judge agrees with reviewers only 71% of the time, and disagreement is concentrated in long, multi-turn conversations where the judge marks answers grounded because they quote some source. Users flag roughly 2% of conversations. Mean time to detect a bad prompt release is currently about four days.
1. How would you improve the judge before trusting it to gate releases?
Read the worked answer
Judge quality is the binding constraint, so fix it before adding sampling volume.
- Diagnose the failure mode, not the score. 71% agreement with disagreement concentrated in multi-turn cases points at the judge evaluating one turn in isolation and rewarding any citation. Change the rubric to require that the cited passage actually supports the specific claim, and give the judge the full turn context plus the retrieved passages.
- Decompose the task. One judge scoring 'groundedness and helpfulness' conflates them. Groundedness should be largely checkable (claim-to-passage entailment); helpfulness is subjective and belongs to a separate, looser rubric.
- Calibrate against humans. Use the 200 human-reviewed conversations as a fixed calibration set; report judge-vs-human agreement per rubric and per conversation length. Target agreement on the automated gate's decision boundary, not overall - a judge that is 95% accurate on clear cases and coin-flip on borderline ones still fails a release gate.
- Version the judge and pin it. A judge silently upgraded by a provider invalidates every historical baseline.
State the limit: a judge at ~85% agreement cannot support a gate that blocks on a 2% quality delta. Size the threshold to the judge's noise.
2. Design a detection path that finds a bad prompt release in hours, not four days.
Read the worked answer
Layer cheap deterministic signals ahead of the expensive judge.
- Deterministic, per-request metrics computed on 100% of traffic: citation present, citation resolves to a real document, answer length distribution, refusal rate, tool-call failure rate, latency. A prompt change that shortens answers moves the length distribution within minutes. These need no judge and no labels.
- Canary comparison: route 5% of traffic to the new config, compare those distributions against the 95% baseline continuously, auto-revert on a shift beyond a pre-set band.
- Judge on sampled traffic as the slower, richer layer - 100/day is fine for trend, useless for same-day detection. Keep it, but stop expecting it to be the alarm.
- User flags as a lagging corroborator, not a trigger; 2% flag rate is too coarse and too slow.
Key judgement: the four-day lag was not a sampling problem, it was relying on the slowest signal as the only signal. Also define what 'bad' means numerically per metric before the release, otherwise the canary has nothing to compare against.
3. Distinguish retrieval failures from generation failures in observability. How does that change remediation?
Read the worked answer
They look identical to the user ('wrong answer') and require opposite fixes.
- Retrieval failure: the correct passage is not in the top-k. Detect by checking, for questions with a known answer, whether the gold passage appears in retrieved context; and in production by clustering low-groundedness answers where retrieval returned nothing relevant.
- Generation failure: the passage was retrieved and the model still misstated it, ignored it, or over-claimed. Detect via claim-to-passage entailment on the retrieved context.
Instrument both separately per request - store retrieved doc IDs alongside the answer. Without that join, every quality regression becomes an argument between the retrieval team and the prompt team.
Remediation differs accordingly: retrieval failures point at chunking, embeddings, top-k, or index freshness; generation failures point at prompt, context ordering, or model choice. A useful rule: if groundedness drops while 'gold passage in top-k' stays flat, it is generation; if the retrieval hit rate drops first, it is retrieval. Track both on the same dashboard so the first one to move identifies the cause.