Offline Groundedness 0.91, Production Thumbs-Down 19%
An insurer's claims-status agent answers 40,000 questions a day. The weekly offline evaluation, using an LLM judge, reports groundedness 0.91 and correctness 0.87 against a 500-question golden set built eight months ago from contact-center transcripts. Since a new state regulation took effect five weeks ago, about 40% of live traffic concerns a document type that appears nowhere in the golden set. Production thumbs-down rose from 6% to 19% over the same period. Two changes shipped three weeks ago: a retriever change (top-k from 8 to 15, new reranker) and a prompt change. Latency p95 rose from 3.1 s to 5.8 s. The team has no human labels for production traffic, and the judge was never validated against human raters.
1. Explain why offline and production disagree, and describe how you would rebuild the measurement so this class of regression is caught automatically.
Read the worked answer
Four independent reasons the numbers diverge, and you should separate them before fixing anything.
Distribution. The golden set was sampled from transcripts eight months ago; 40% of today's traffic concerns a document type absent from it. Offline scores are computed on a population that mostly no longer exists, so 0.91 is a number about the past.
Unvalidated judge. No agreement study against humans was ever run. Judges systematically favor fluent, confident, verbose answers, which inflates groundedness precisely when retrieval returns nothing and the model improvises. A judge that has never disagreed with a human is a judge whose score you cannot interpret.
Missing outcome class. The correct behavior for out-of-corpus questions is refusal or escalation. If the rubric only scores answer quality on answerable questions, correct refusals are unscored and wrong answers on unanswerable questions go unpunished.
Signal mismatch. Thumbs-down also captures latency - p95 nearly doubled - so part of the 19% is user frustration, not answer quality. If you optimize only on accuracy, you will not move that number.
Rebuild: sample 400-500 recent production questions, stratified by intent, state, and - critically - whether the governing document is in the corpus. Human-label 150-200 of them, including refusal correctness. Measure judge-human agreement, then regenerate the golden set from production on a rolling cadence, and report retrieval metrics separately from generation metrics so missing evidence and bad synthesis are distinguishable.
2. Isolate which of the two shipped changes caused the regression. Describe the experiment.
Read the worked answer
Replay the same inputs under all four combinations. Freeze a set of 200 recent production queries with human labels. Run old retriever + old prompt, new retriever + old prompt, old retriever + new prompt, and new + new, against the identical chat model version. Anything that varies between arms other than retriever and prompt invalidates the comparison - the model version must be pinned.
Read the interaction, not just the main effects. A regression that appears only in the new+new arm means the prompt was written for the old evidence format and does not survive 15 chunks with reranking. A regression present whenever the new retriever is used points at the reranker or at the larger k injecting conflicting chunks.
Segment the result. Split by whether the new-regulation document exists in the corpus. If the regression concentrates there, neither change caused it - the corpus has no coverage for the new document type, the retriever returns weak matches, and the model fabricates. The retriever change may simply have made a pre-existing coverage gap more visible by returning more confident-looking near-misses.
Attribute latency directly. k=8 to k=15 plus a reranker pass plausibly accounts for ~2.7 s of added p95; confirm with per-stage timings in traces rather than assuming, because the prompt change may also have lengthened output.
Then shadow in production on a small slice before full rollout, since replay cannot capture user phrasing drift or cache effects.
3. Design the ongoing production monitoring: what do you log, what alerts fire, and how do you avoid alert fatigue?
Read the worked answer
Log per request: query text (with PII handling), retrieved chunk ids with scores, whether the governing document was present, final answer, model and prompt versions, index version, config hash, per-stage latency, and user feedback. Inference tables plus tracing on the serving endpoint give this without instrumenting the application twice.
Metric layers, because they fail differently:
- Retrieval: empty-result rate, recall against known-answer probes, score distribution shift. Empty-context rate creeping above a few percent predicts hallucination before thumbs-down moves.
- Generation: sampled judge scores on live traffic, refusal rate, citation validity.
- Experience: thumbs-down rate, p95 latency, escalation-to-human rate.
- Data drift: distance of live query embeddings from the golden-set distribution, plus coverage checks against newly published document types.
Alerts that earn their pager: thumbs-down above an absolute threshold for three consecutive days; empty-context rate spike; p95 beyond SLO; judge score drop beyond a confidence bound week over week; retrieval miss on a canary probe set that runs every few minutes. Tie every alert to the config hash so you can name the deploy.
Avoid fatigue by alerting on sustained deviation and on leading indicators (empty context, probe misses) rather than on every thumbs-down, and by routing drift signals to a weekly review queue rather than an on-call page.
Close the loop: 50 sampled conversations reviewed by a human weekly, fed back into the golden set and into judge calibration. Without that, the judge drifts and the monitor watches itself.