HR Policy RAG: Relevance and Citation Failures After a Model Upgrade Proposal
An HR shared-services team deployed a retrieval-augmented assistant over 18,000 policy documents, including country-specific leave rules, benefits summaries, and union agreements. Documents change weekly; 2,400 pages were updated last quarter. The index was last rebuilt 11 days ago, and the retriever has no effective-date or jurisdiction filter. A spot audit of 300 answers found 31% were rated not fully relevant, 12% had no usable citation, and 7% quoted a superseded policy. The team's proposed fix is to switch from a 7B to a 70B generator and raise top-k from 4 to 12 chunks. Average response time is already 6.8 seconds, and the team has a 9-second p95 target. Legal requires that every answer cite the controlling policy and that the assistant refuse when sources conflict. The team has no offline evaluation set and releases prompt changes directly to production.
1. The team proposes a larger generator and higher top-k. Evaluate whether that addresses the audit findings, and propose a diagnosis plan.
Read the worked answer
The proposal treats a retrieval and freshness problem as a generation-capacity problem. A larger generator may improve fluency but will not fix missing citations or superseded policy if the retriever returns stale or irrelevant chunks; raising top-k to 12 adds noise and latency, likely pushing p95 past 9 seconds. Diagnosis: sample 100 failed answers and label the failure stage: query understanding, retrieval recall, ranking, chunking, generation, or citation. Check whether the controlling document was in the index and within the top 20. Inspect effective-date metadata and whether the retriever filters by country and employment type. Measure recall@k and MRR before changing the model. Fixes likely include hybrid keyword plus vector search, metadata filters for jurisdiction and effective date, a reranker, smaller chunks with section headers, and a freshness pipeline that rebuilds changed documents within 24 hours. Keep the 7B model until retrieval is strong; then test a larger model only on groundedness and citation accuracy. Trade-off: reranking adds roughly 300-600 ms but usually improves precision more than top-k expansion.
2. Design an evaluation harness and release gate for weekly policy updates.
Read the worked answer
Build a golden set of 400 question-answer pairs with controlling policy IDs and effective dates, plus 50 conflict cases. Separate metrics: retrieval recall@10 for the controlling document, citation validity, answer groundedness, conflict refusal rate, and latency. Use LLM-as-judge only with human calibration on 100 samples; track agreement. Release gate: no deploy if recall@10 <0.95, citation validity <0.98, or groundedness <0.95 on the golden set; conflict cases must refuse 100%. On every weekly document update, run a delta evaluation on affected policies before index promotion. Version prompts, retrievers, and indexes together; canary 5% of traffic and compare to control. Rollback if citation validity drops more than 2 points or refusal rate on answerable questions rises more than 3 points. This separates content freshness from model behavior and prevents a prompt change from masking an index defect.
3. How would you implement refusal and conflict handling so that legal requirements are met?
Read the worked answer
Implement conflict detection at retrieval: group chunks by policy ID and effective date; if two active documents give different answers for the same jurisdiction and employee type, return a conflict response with both citations and route to HR. The generator must be constrained to answer only from retrieved chunks and must include document ID and effective date. If no chunk exceeds a relevance threshold, abstain. Add a post-generation verifier that checks each claim against cited text; if unsupported, remove or refuse. Log refusals for review. Do not let the prompt alone enforce this; use deterministic checks. The trade-off is more refusals, but legal exposure from a confident wrong answer is higher than the cost of an escalation.