The RAG Assistant That Cites Confidently Wrong Policy
An HR shared-services team deploys a retrieval-augmented generation assistant over 9,000 internal policy documents to answer employee questions about leave, expense, and conduct. In a 300-question evaluation, 88% of answers were rated 'helpful' by reviewers, but 6% contained a policy statement contradicted by the source document, and 4% cited a document that exists but does not contain the claim. The assistant answers in fluent, confident prose with inline citations. The team's proposed fix is to add 'be accurate and only cite real documents' to the system prompt and to ship to all 12,000 employees next sprint, with a disclaimer banner. Legal has flagged that employees may rely on wrong leave answers.
1. Evaluate the proposed fix and the ship decision.
Read the worked answer
The prompt fix is insufficient; shipping to all employees next sprint is not justified.
Why the prompt won't work: instruction-following cannot guarantee factual grounding. The failure modes here - unsupported claims and miscited-but-real documents - are retrieval and attribution failures, not tone failures. A model told to 'be accurate' has no mechanism to verify a citation actually supports a sentence. Adding a disclaimer shifts risk to the employee and does not reduce the error rate.
Quantify the harm: 6% contradicted + 4% miscited = 10% of answers defective on a 300-question eval. At 12,000 employees, even a few questions each per month yields hundreds of wrong leave/expense answers. Legal reliance risk is real, and leave policy errors can create compliance exposure.
What to do instead: (1) treat citations as claims to be verified, not decoration; (2) gate high-stakes categories (leave, conduct) behind human review or a curated FAQ; (3) pilot with a small group and measure.
2. Propose the technical and process changes that would most reduce the 6% and 4% error rates, and how you would measure success.
Read the worked answer
Technical changes
- Quote-then-answer: require the model to extract verbatim supporting passages first, then compose the answer only from those passages. Refuse if no passage supports the claim.
- Post-hoc citation verification: a second pass checks each cited sentence against the retrieved chunk (entailment check). If it fails, the answer is withheld or flagged.
- Retrieval quality: evaluate recall of the retriever separately. If the right document isn't retrieved, no prompt fixes the answer. Chunk policies with metadata (effective date, supersedes) so stale versions don't surface.
- Abstention path: when confidence is low or sources conflict, the assistant says so and routes to a human - explicitly rewarding abstention over guessing.
Process changes
- Human review queue for leave and conduct answers during pilot.
- Versioned policy corpus with effective dates; retire superseded docs.
Measurement
- Re-run the 300-question eval with a rubric that separately scores groundedness (every claim supported by a cited passage) and citation correctness. Target: contradicted-claim rate <1%, miscitation <1% before broad release.
- Track production thumbs-down and escalation rate as leading indicators.