The Support Assistant That Invented a Refund Policy
A subscription software company deploys a generative AI assistant that answers customer questions using a foundation model with a retrieval layer over its help-center articles. In the first month, the assistant handles 40,000 conversations. Agents escalate 1,200 of them. In 300 of those escalations, customers were told about a '30-day no-questions refund' that does not exist; the actual policy is 14 days with conditions. The retrieval layer returned the correct policy article in 85% of those cases, but the model still produced the wrong answer. The company also notices the model gives confident, fluent answers to questions about features that have never existed. Leadership wants to know whether to blame the retrieval system or the model, and what to change before scaling.
1. Given that retrieval returned the correct article in 85% of the failing cases, analyze the likely source of the incorrect refund answers and propose a fix.
Read the worked answer
If the correct policy article was retrieved in 85% of failing cases but the model still output a nonexistent policy, the retrieval layer is not the primary culprit for those cases. The model is not grounding its answer in the supplied context: it is generating from parametric knowledge and prior patterns, which is hallucination. Fluent, confident fabrication is a hallmark of this failure mode, and it is consistent with the model answering 'what a refund policy usually is' rather than 'what this company's policy says.'
The fix has two parts. First, strengthen grounding at generation time: instruct the model to answer only from retrieved passages, cite the passage used, and return a fallback such as 'I don't have that information - let me connect you' when the context does not contain the answer. Second, verify after generation: a lightweight check that the answer's key claims (refund window, conditions) appear in the retrieved text, with automatic escalation on mismatch.
For the remaining 15% where retrieval failed, improve the retrieval layer separately: better chunking, metadata filters on policy documents, and evaluation of retrieval recall. The two problems need separate metrics so the team can tell whether a change fixed grounding or fixed retrieval.
2. Design a measurement plan that would have caught this problem before full rollout, including what to log and how to gate the launch.
Read the worked answer
The team needs evaluation that targets faithfulness, not just fluency or customer satisfaction. A practical plan:
Build a labeled evaluation set of real customer questions, with the correct policy answer and the source article for each. Include adversarial questions about policies that do not exist, to test whether the model refuses or fabricates.
Measure separately: retrieval recall (did the right article appear in context?), answer faithfulness (does every claim trace to the retrieved text?), and answer correctness. These separate numbers tell the team which component to fix.
Log every conversation with the retrieved passages, the model output, citations, and any guardrail or verification flags. Sample escalated conversations weekly and review for fabrication patterns.
Gate the launch on thresholds: for example, faithfulness above a target on the evaluation set, zero fabricated-policy answers in the adversarial set, and a monitored escalation rate with alerts when it moves.
Run a limited canary or shadow deployment first, comparing assistant answers against human agent answers on the same questions, and expand only when the measured fabrication rate is acceptable.