The 89% Eval That Became 54% in Production
A team assembles 480,000 instruction-response pairs by scraping public forums and support boards. Exact-match deduplication removes 38,000, leaving 442,000 for training. They hold out 2,000 pairs as the evaluation set and fine-tune a 7B model with LoRA. On that eval set the tuned model wins 89% of pairwise comparisons against the base model, judged by a strong LLM. Two weeks after launch, human reviewers rate 300 sampled production responses: 54% are acceptable. Production prompts have a median length four times the eval set, 60% are multi-turn, and many contain product jargon that appears nowhere in the scrape. The eval set is single-turn with a median prompt of 42 tokens.
1. The team concludes the model regressed after deployment. Evaluate that conclusion and identify the most likely explanations for the 89% vs 54% gap.
Read the worked answer
The regression hypothesis is the least likely explanation. Nothing changed in the weights between the eval run and launch, so the gap is almost certainly a measurement artifact plus a distribution gap.
- The two numbers measure different things. 89% is a relative win rate against a weak base model. 54% is an absolute acceptability rate against a rubric. A judge can prefer an answer to a bad baseline while the answer is still wrong or unusable. These are not comparable and should never have been placed on the same axis.
- Near-duplicate contamination. Only exact matches were removed. Scraped forum corpora have heavy paraphrase duplication; if a meaningful share of eval items has a training neighbor at high embedding similarity, the model is being tested on material it memorized.
- Distribution shift. Four-times-longer prompts, 60% multi-turn, unseen product jargon. A single-turn 42-token eval cannot expose any of those failures.
- Holdout drawn from the same scrape. The eval inherits the scrape's topic mix and style, so it cannot represent real tickets even in principle.
- Judge bias. An LLM judge rewards fluency, length and confident tone; humans rate against task completion.
The defensible statement is that the eval was not predictive of the deployment population, not that the model got worse.
2. Design an evaluation protocol that would have caught this before launch, and specify how you would quantify the contamination rather than assert it.
Read the worked answer
Measure contamination, do not assume it. For each of the 2,000 eval items, compute max cosine similarity against the training set (embedding index) and max 13-gram overlap. Report the share above a threshold such as 0.90 similarity or 50% n-gram overlap. Then re-run the win rate on the uncontaminated subset only. The difference between full-set and clean-subset scores is the contamination inflation, stated as a number. If the clean subset is too small to be decisive, say so instead of rounding to a claim.
Rebuild the eval from the deployment population. Sample real tickets, not scraped pairs. Stratify by turn count (single vs multi), prompt length bucket, product area and whether jargon appears. Size each slice to support the decision you need to make - a slice with 40 items supports a smell test, not a ship/no-ship call.
Score on an absolute rubric, not only pairwise vs baseline: task completed, factually supported, actionable. Keep a pairwise comparison as a secondary signal.
Add a held-out regression set with at least 200 multi-turn conversations, since that is 60% of traffic and was 0% of the old eval.
Report per slice, always. A single aggregate number hides exactly the shift that caused this failure. Publish the slice table with the launch decision, and re-run it on a weekly traffic sample after launch so the same blind spot cannot reopen.