The Retrieval System That Cites Beautifully and Answers Wrong
A legal-research assistant answers questions over a 400,000-document contract corpus. Pipeline: chunk at 512 tokens with 50-token overlap, embed with a general-purpose text embedding model, retrieve top-8 by cosine similarity, and pass chunks to Claude with the instruction "answer using the provided context." Offline eval on 150 hand-written questions shows 88% answer accuracy. In production, a partner reports three answers that cite real clauses but reach the wrong conclusion - the retrieved chunk was from a different contract with near-identical boilerplate. You also notice that questions about defined terms ("what does 'Effective Date' mean in this agreement?") retrieve poorly. Latency is fine; cost is not a concern. The team proposes switching to a larger embedding model and increasing top-k to 20.
1. Evaluate the team's proposal. What does it fix, and what does it not fix?
Read the worked answer
The proposal addresses the wrong failure mode and leaves the dangerous one untouched.
What a bigger embedding model might help: defined-term queries, marginally - a stronger model may better separate "Effective Date" from generic date language. But that failure is more likely a chunking problem: a defined term is defined in one place and used everywhere, and a 512-token chunk that contains the use but not the definition will never retrieve the definition no matter how good the embedder is.
What it does not fix - and this is the serious one: the cross-contract contamination. The bug is that retrieval has no scoping constraint. The query is about contract A, but the index is global, so boilerplate from contract B scores high. A larger model and top-20 make this worse: more chunks, more near-duplicate boilerplate, more chances to pull the wrong contract's clause. The model then cites a real clause - from the wrong document - which is the most dangerous failure class because it looks verifiable.
The actual fix is metadata filtering, not similarity. Every chunk must carry contract_id (and ideally section, party, execution date). Retrieval must be constrained to the contract(s) in scope, resolved from the user's question or session. That's a hard filter applied before ranking, not a soft signal the embedder is asked to infer.
Secondary fixes: chunk by document structure (clause/section boundaries) rather than fixed 512 tokens, and add a defined-terms index that maps term → definition chunk → contract. Only after scoping is fixed is it worth revisiting the embedder.
2. The offline eval said 88%, but production is failing. What is wrong with the eval, and how would you rebuild it?
Read the worked answer
The eval measures the wrong thing against the wrong distribution.
Problems:
- It scores the answer, not the retrieval. If you only check the final answer text, a confident wrong answer that cites a real clause can pass a lenient grader. You need retrieval-level metrics (did the gold chunk appear in top-k?) separate from answer-level metrics.
- No cross-contract adversarial cases. 150 hand-written questions almost certainly each target one obvious contract. The production failure is confusable contracts - near-duplicate boilerplate. The eval set contains none of the thing that breaks.
- No defined-term cases. A whole query class is untested.
- Likely leakage or leniency in grading. Hand-written by the team, possibly graded by the same model that answers. 88% is suspiciously tidy.
Rebuild:
- Build a retrieval eval: for each question, label the gold chunk(s). Report recall@k and MRR. This isolates pipeline from generation.
- Add a contamination suite: pairs of near-identical contracts; ask about contract A and assert no chunk from B is retrieved or cited. This is the regression test for the production bug.
- Add a defined-term suite: term-in-contract → definition chunk.
- Add an answer eval with a rubric that checks (a) conclusion correctness and (b) that every cited clause belongs to the in-scope contract. Use a different model or human spot-check for grading, and track inter-rater agreement.
- Stratify by difficulty and report per-stratum, not a single blended number - a blended 88% hid a 0% on the class that matters.