HR Policy Assistant Retrieval and Grounding Failures
A people-operations team launches an internal policy assistant over 1,200 HR PDFs averaging 14 pages each. Documents are split into 512-token chunks with 50-token overlap, embedded with a general-purpose embedding model, and the top five chunks are passed to a hosted LLM at temperature 0.7 with an 8K context window. In a 300-question evaluation set, retrieval recall@5 is 0.61. In a separate audit of 150 sampled answers, 22% are judged unsupported by the chunks that were retrieved, and users report the assistant frequently cites a superseded 2021 leave policy instead of the 2024 version. Long compensation tables are also being split mid-row, producing plausible but wrong figures. The team has four weeks before a company-wide launch and can change chunking, retrieval, prompting, and model settings, but cannot renegotiate the LLM vendor contract.
1. Separate the likely causes of the 22% unsupported answers from the causes of the superseded-policy citations, and explain how you would diagnose which is which.
Read the worked answer
The superseded-policy citations are almost certainly a retrieval and metadata failure, not a generation failure. Both versions sit in the same index with similar embeddings, and nothing in the pipeline filters by effective date or superseded status, so the retriever can return the 2021 text and the model faithfully summarizes what it was given.
The 22% unsupported rate mixes at least two causes: chunks that never contained the answer, and chunks that did but were ignored or misread. To separate them, take the 150 audited answers and mark, for each, whether a chunk supporting the correct answer appeared in the top five. If yes but the answer is unsupported, the problem is grounding, prompt adherence, or context position; if no, it is retrieval. I would also check whether the model cited a retrieved chunk at all, and whether the correct chunk appeared lower in the ranking. The table issue is a chunking failure visible in both retrieval and generation: splitting rows destroys the row-label relationship, so no model can recover the figure reliably.
2. Propose a prioritized remediation plan for the four-week window, explaining which changes you expect to have the largest effect.
Read the worked answer
Order by expected impact per engineering week.
- Add metadata and filtering: effective date, superseded flag, policy owner, and audience. Exclude superseded documents by default and let the query specify a historical version. This directly removes the stale-policy failure and is cheap.
- Fix chunking: split on headings and sections, keep tables intact as single units or convert them to structured text, and use parent-child retrieval so a small chunk is matched but a larger section is passed to the model. This addresses the compensation-table errors.
- Improve retrieval: hybrid BM25 plus dense retrieval, retrieve top 20 and rerank to top 5 with a cross-encoder, and add query rewriting for policy names and synonyms. Recall@5 of 0.61 is the ceiling on answer quality, so this is the highest-leverage change after filtering.
- Tighten generation: lower temperature to 0.1–0.2, require a citation for every claim, instruct the model to abstain when the context is insufficient, and place the most relevant chunks first.
- Reindex and rerun the 300-question evaluation after each step so the effect of each change is measured rather than assumed.
3. Design the evaluation that would convince you the assistant is launch-ready.
Read the worked answer
I would build a gold set of at least 300 questions, expanded to cover every policy area, document type, and the table-heavy compensation content. Each item needs a reference answer, the supporting chunk IDs, the correct policy version, and a label for answerable versus unanswerable.
Metrics should be separated by stage. Retrieval: recall@5, recall@20, MRR, and stale-version rate. Generation: answer correctness, faithfulness to the retrieved chunks, citation accuracy, and correct abstention on unanswerable questions. I would calibrate an LLM judge against a human-audited sample of at least 100 answers and report agreement, rather than trusting the judge blindly.
Acceptance thresholds should be explicit, for example recall@5 ≥ 0.90, faithfulness ≥ 0.95, stale-version rate of 0, and abstention ≥ 0.90 on unanswerable items, with no slice below a defined floor. I would also run the suite as a regression gate on every index or prompt change, canary the launch to one department, and track user-reported errors and thumbs-down rates during the canary.