The Judge Says 92%, Users Say Thumbs-Down

The judge measured the wrong thing. "Is the answer supported by the retrieved context?" is a groundedness question. An answer that faithfully quotes a policy superseded last quarter scores 1.0. The team built a metric that is blind to every failure they actually shipped, then used it as a launch gate.

Bucket classification - none of these are generation-quality failures:

Bucket	Count	Root cause	Layer
Outdated policy	30	Stale corpus; superseded documents still retrievable and rank highly because they are a lexical match	Data/indexing
Cites inaccessible doc	12	Retrieval runs under a shared identity that can see everything; per-user entitlements are not applied	Permissions
Over-refusal	8	Prompt or coverage - possibly the guardrail prompt is over-broad, or the answer exists in a document that was never ingested	Prompt/coverage

Only the third bucket could plausibly involve model behavior, and its cause is undetermined. The majority (42 of 50) are infrastructure defects. A model that is smarter would still quote the stale policy and still cite the restricted document.

Also note what the eval set was: 60-ish curated questions, not production questions. It likely never sampled "did policy X change recently" or "what can a contractor in region Y see." Coverage gap, not calibration gap.

No. It would make the dominant failure worse.

Fine-tuning changes behavior - tone, format, citation style, refusal calibration, output structure. It does not install a fact that changed three weeks ago, and it cannot make an ACL problem disappear. Worse:

• The 5,000 historical Q&A pairs were generated against the old corpus. Training on them bakes stale policy into the weights, as ground truth, with no retrieval trace to attribute or correct later. You would convert a fixable corpus bug into a model that is confidently wrong and cannot be fixed without a retrain.
• Historical pairs almost certainly contain employee-specific data and no correctness labels. Training before labeling is training on noise.
• Fine-tuning does not address the ACL bucket at all; that is a permissions problem in the retrieval path.

When fine-tuning is the right answer: the evaluation shows residual errors are stylistic or structural - inconsistent citation format, unreliable JSON, refusal phrasing that oscillates, or a smaller model needs distillation to hit a latency/cost target. That is a behavioral gap with an eval that proves it.

What to do instead, in priority order: expire/re-index superseded policies and filter on effective_date; enforce document permissions at retrieval time; then examine the 8 refusals with a labeled sample to decide whether the fix is prompt, coverage, or genuinely the model.

Replace the single judge with multiple independent, narrowly scoped scorers - each one answerable, each one separately reported:

1. Groundedness - is the answer supported by the retrieved context? (Keep it; it is necessary and insufficient.)
2. Correctness - does it match the authoritative current policy text? Requires a human-labeled reference, not the context itself.
3. Freshness - does the cited document have an effective_date at or after the most recent change on that topic? A rule-based check, not a judge call.
4. Permission compliance - every cited chunk has visibility the asking role can access. Deterministic.
5. Retrieval recall@k - did the chunk that should answer the question appear at all? Separating retrieval from generation is what tells you which layer broke.
6. Refusal appropriateness - on a set containing both answerable and genuinely out-of-scope questions, penalize false refusals and unsafe answers independently.

Build the golden set from production, not from imagination: real questions, weighted by frequency, plus adversarial cases and a deliberate slice of questions about recently changed policies and restricted documents. Label with humans; sample the judge against those labels.

Slice everything. By policy domain, question type, document age, and user role. A 92% aggregate with a 40% score on "changed in last quarter" is a launch blocker that the aggregate concealed.

Instrument the judge itself. Track judge–human agreement per slice; retire or repair any scorer whose agreement drops. Run a continuous evaluation on a sampled slice of production traffic and route low-scoring or low-confidence answers to a human review queue for calibration. A metric nobody re-validates is a number, not a measurement.