Weekly Prompt Shipments With a Judge That Likes Long Answers
A six-person team ships prompt and model changes weekly to a document-QA assistant used by 1,200 internal staff. Regression testing is manual: someone asks ten questions and eyeballs the answers. Last quarter three regressions reached users - citation links stopped resolving, answers to 'how do I request leave' became 40% longer with no added content, and a model swap dropped accuracy on table questions. They now have 400 golden examples, but only 120 were written by a domain expert; the rest were model-generated and spot-checked. Their proposed LLM-judge setup reached 92% agreement with a human on 50 examples, yet the judge systematically scores longer answers higher. CI must finish in 15 minutes, and the team has budget for roughly 40,000 judge tokens per run.
1. Design the evaluation suite: layers, sample sizes, and CI gating, within the 15-minute and 40,000-token constraints.
Read the worked answer
The 15-minute ceiling forces you to decide what runs when.
Layer 1 - deterministic, every commit, ~3 minutes. About 40 exemplar inputs with assertions that need no judge: citation URLs resolve to real document IDs, JSON schema valid, non-empty answer, length within band. Note that all three shipped regressions were catchable here - the broken link and the length blowup needed no model at all, which tells you the team was missing checks, not intelligence.
Layer 2 - model-graded, every pull request, ~10 minutes. 150 stratified examples with at least 30 table questions. Budget arithmetic matters: 150 × (600 input + 120 output) ≈ 108k tokens, well over the 40k allowance. So either cut to ~60 examples, or send only the retrieved passage (≈150 tokens) rather than full documents - 150 × 270 ≈ 40k, which fits. Pick one deliberately and write the choice down.
Layer 3 - full 400 plus human spot-check, nightly. Never in CI.
Gates: Layer 1 must be 100%, hard block. Layer 2 blocks on any aggregate metric down more than 3 points versus the last accepted baseline, or any single category down more than 5. Same held-out set every run; version it alongside the prompt.
2. The judge agrees with humans 92% of the time on 50 examples but rewards verbosity. Diagnose this and describe how you would fix and validate the judge.
Read the worked answer
92% agreement on 50 examples is a rumour, not evidence. There is no chance-corrected statistic, no error-direction analysis, and the sample is too small to distinguish 92% from 85%.
Why it rewards verbosity: pointwise rubrics containing words like 'thorough' or 'clear' correlate with length, and judges tend to prefer longer, more elaborate outputs. That is precisely why the 40%-longer leave answers passed review.
Remediate:
- Judge pairwise, not pointwise. Compare candidate against baseline with position-swapped trials; pairwise comparisons control for length far better than absolute scores.
- Grade checkable criteria first: citation resolves, required facts present, no claim outside the retrieved source. Grade facts, not prose.
- Add an explicit anti-verbosity rule to the rubric: do not reward length; a correct three-sentence answer beats a correct ten-sentence one; penalize unsupported content.
- Penalize or normalize length in the score if verbosity persists.
Validate properly: use 100 expert-labeled examples including 30 adversarial pairs where the shorter answer is better and 30 where the longer one is. Report agreement and the judge's error direction on those pairs. If it fails the traps, it fails - regardless of the 92%. Re-validate whenever the judge model, rubric, or prompt changes, and version the judge like any other artifact.
3. Only 120 of the 400 golden examples are expert-written. How do you handle the other 280?
Read the worked answer
Treat the 120 as gold and the 280 as coverage data with an unverified prior.
- Never gate on unverified labels. Model-generated examples encode the generating model's errors; a suite built from them rewards the behaviour you are trying to detect. They are useful for coverage questions (do we have a table question, a multi-hop question, a citation question?), not for pass/fail decisions.
- Prioritize labelling by consequence and by disagreement. Rank the 280 by: document types implicated in past incidents (tables first, given the model swap); items where two judges or two models disagree; items where the system is confidently wrong. Label 40–60 of those before the next release.
- Run an active labelling loop. Every production complaint becomes a candidate example, reviewed by the domain expert within a week and frozen into the set with a version tag. That is how the set stays relevant without ballooning.
- Report confidence honestly. With 120 expert examples, an 80% accuracy figure carries roughly ±8 points at 95% confidence. Put that number in the release note and mark the remaining metrics 'provisional'. Claiming precision you do not have is how the next regression ships unnoticed.