Retrieval-Augmented Policy Assistant With Stale Citations
Northwind Legal Ops deploys a retrieval-augmented assistant over 40,000 internal policy documents. The corpus is chunked at fixed 380-token windows with 40-token overlap, producing 2.1 million chunks. The retriever uses a general-purpose embedding model and cosine top-k=5; the generator receives the five chunks and an 8,000-token context window. In a 500-question gold evaluation, retrieval recall@5 is 0.62 because gold passages are often split across chunk boundaries that cut tables and definitions. Answers cite a 2019 travel policy as current even though a 2024 revision exists; the 2024 document is retrieved only 31% of the time when the question mentions 'current'. Human raters mark 14% of answers as unsupported by the cited context. Queries often combine two asks, such as 'What is the meal cap for Singapore and does it differ for overnight travel?' The team proposes increasing top-k to 20 and buying a larger generator.
1. Diagnose whether the primary failure is retrieval or generation. Explain what evidence you would use to decide, and why the proposed top-k increase and larger generator may not help.
Read the worked answer
Recall@5 of 0.62 over 2.1 million chunks points to retrieval before generation. Fixed 380-token windows cut tables and definitions, so the gold evidence is split; the generator then sees a partial definition and fills gaps. The 2019 citation is partly a metadata failure: the retriever has no effective-date filter, and 'current' is not a strong semantic cue when both versions are near-duplicates. The 14% unsupported rate is a generation/grounding failure amplified by retrieval; with poor chunks, top-k=5 gives plausible but incomplete context. Increasing top-k to 20 without reranking adds noise and may worsen grounding, while a larger generator cannot cite evidence it never received. Diagnose with a retrieval-only test: for each gold question, measure recall@k, MRR, and whether the gold span is fully contained in a retrieved chunk. If full-span recall is low, fix chunking first. If the correct document is retrieved but the answer remains unsupported, fix prompting and citation constraints.
2. Propose a concrete retrieval and chunking redesign. Include how to handle versioned policy and multi-part queries.
Read the worked answer
Adopt structure-aware parent-child chunking: split on headings, definitions, and table boundaries; index small child chunks of 150–250 tokens for retrieval but return the parent section of up to 1,200 tokens to the generator. Attach metadata for effective_date, supersedes, doc_type, and jurisdiction. At query time, parse relative-time intent such as 'current' or '2024' and filter to the latest effective version, or apply a strong recency boost. Use hybrid retrieval: BM25 for exact policy names and numbers, plus dense embeddings, fused with reciprocal rank fusion, then a cross-encoder reranker over the top 50 to return 8 parent sections. For multi-part queries, decompose into sub-questions, retrieve per sub-question, deduplicate, and require the generator to answer each part with a citation. This addresses the meal-cap and overnight-travel question better than a single top-k. Keep the 8,000-token context but prioritize parent sections; do not simply raise top-k to 20. The larger generator should be tested only after retrieval full-span recall exceeds 0.90.
3. Design an evaluation plan that would catch the stale-policy and grounding failures before launch. Specify metrics, slices, and a guardrail.
Read the worked answer
Build a 500-question gold set with span-level evidence labels and version metadata. Report retrieval recall@5, full-span recall@5, MRR@10, and nDCG@10. For generation, measure groundedness (every claim supported by a cited span), answer correctness, citation accuracy, and stale-policy rate. Use a blinded two-annotator protocol with adjudication and report Cohen's kappa. Add a 'current policy' slice and a multi-part slice because aggregate scores hide these failures. Measure stale-citation rate specifically for the 2019/2024 issue. Run an ablation: chunking only, metadata filter only, hybrid plus reranker only, and the full pipeline. Guardrail: no slice may fall below 0.90 groundedness, and stale-citation rate must drop below 2%. Offline evaluation is necessary but not sufficient: canary the assistant on 200 internal queries with human review, and log retrieval provenance so a wrong citation can be traced to the chunk, rank, or filter.