RAG Assistant Cites the Wrong Version of a Policy
An internal HR assistant answers employee questions using retrieval-augmented generation over 12,000 documents. Documents are chunked at 512 tokens with no metadata beyond source filename. Users report that answers about parental-leave policy sometimes cite a 2019 version instead of the current 2024 version, and that answers to multi-part questions ("what's the leave allowance and how do I apply?") cover only one part. Retrieval returns the top 5 chunks by embedding similarity. The corpus contains many near-duplicate documents from annual revisions. Latency budget is 3 seconds; current p95 is 1.8 seconds.
1. Explain why embedding similarity alone retrieves stale policy versions, and propose a retrieval change that fixes it without exceeding the latency budget.
Read the worked answer
Embedding similarity measures topical closeness, not recency or authority. A 2019 parental-leave doc and its 2024 replacement are near-identical in meaning, so they score almost the same - and if the 2019 text is chunked slightly differently it can even rank higher. The retriever has no signal that one is superseded.
The fix is metadata-aware retrieval with a recency/authority filter, not a better embedding model:
- Add structured metadata at ingest:
doc_type,effective_date,superseded_by,status(current/archived). - At query time, filter to
status=currentbefore or alongside the vector search (most vector stores support metadata pre-filtering). - If the store can't pre-filter cheaply, retrieve top 20 by similarity, then rerank with a cheap rule: drop superseded docs, prefer latest
effective_datewithin the samedoc_type.
This adds negligible latency - a metadata filter is cheaper than widening top-k, and a rule-based rerank over 20 candidates is milliseconds. It stays well inside the 3s budget. The key insight is that the staleness is a data modeling failure, not a retrieval-algorithm failure; no amount of embedding tuning fixes missing version metadata.
2. Address the multi-part-question failure. What would you change, and how would you test that the change generalizes?
Read the worked answer
The multi-part failure is a query-construction problem: a compound question produces one embedding that averages two topics, so it lands between them and retrieves chunks for whichever dominates. Top-5 from a single blended query can't reliably cover both.
Change: decompose the query before retrieval. Either (a) a cheap pre-step where Claude splits the question into sub-questions, retrieves per sub-question, and merges/dedupes chunks, or (b) issue the original query plus per-sub-question queries and union results. Option (a) costs one extra model call; budget it against the 1.2s of headroom. Also instruct the answer step to explicitly address each sub-question and flag any it couldn't source.
Testing for generalization: build an eval set of compound questions with known required sub-answers and check coverage - does the final answer contain a correct, cited response to each part? Include adversarial compounds where one part is unanswerable from the corpus to confirm the model says so rather than hallucinating. Track per-part coverage rate, not just overall answer quality, and hold a regression set of single-part questions to ensure decomposition doesn't degrade simple cases or blow the latency budget.