The RAG Assistant That Cites a Retired Article

Replay each flagged case and record the retrieved chunks alongside the answer, then sort into three buckets:

• Retrieved but stale or irrelevant - the archived article is sitting in the context. That is an index and metadata problem; no prompt can fix it.
• Correct content in context, output contradicts it - a generation problem. Prompt and decoding parameters apply.
• Nothing relevant retrieved - a recall problem: chunking, embedding, or top-k.

A decisive test: hand-feed the gold article into the context and re-ask. If the answer becomes correct, retrieval is the bottleneck, not the prompt.

Expected triage: the stale citation is index-side (no recency filter, archived docs still in the corpus, ticket chatter outranking policy text). The invented refund window is both - no relevant chunk retrieved and no instruction permitting a refusal. Truncation is purely max_tokens. The 14% flag rate splits across all three.

The judgement call is resisting the 'improve the prompt' framing: at most two of the four symptoms are prompt-shaped.

Retrieval and index (addresses stale citations and part of the hallucinations)

• Filter archived and superseded documents at index time via metadata; the model should never see a retired article.
• Split articles into sections with heading-aware chunking and overlap, rather than fixed windows.
• Separate the ticket index from the article index, or weight articles higher; four years of agent chatter is not policy.
• Retrieve more candidates and rerank, rather than trusting top-3 cosine.

Decoding (truncation, and stray creativity)

• max_tokens to roughly 600 for the expected answer length, with an instruction to stay brief.
• temperature 0.1-0.2 and top_p around 0.9. This is grounded extraction, not creative writing; 0.9/1.0 invites invention.

Prompt (refusal behaviour and citations)

• Ground explicitly: answer only from the provided context; if it is absent, say so and route to a human.
• Require a citation of the chunk id supporting each claim, and state the effective date of the policy being cited.
• Add two or three few-shot examples, including one correct refusal and one concise answer.

Be explicit that truncation is a parameter fix and staleness is an index fix - leaving them to the prompt guarantees they persist.

Build a golden set before changing anything. 150-250 realistic questions stratified by intent (billing, SSO, API limits, data export), plus roughly 30 questions whose correct answer is refusal because the knowledge base does not cover them. Without the unanswerable slice you cannot tell improvement from a model that has learned to sound confident.

Metrics that match the symptoms

• Groundedness: every claim traceable to a cited chunk. Automatable with a judge model, spot-checked by humans.
• Citation validity: does the cited article exist, and is it current? This directly tracks symptom one.
• Refusal correctness on the unanswerable slice.
• Answer relevance and length compliance (tracks truncation).
• Containment and escalation rate in production, which is what the business actually feels.

Process. Re-run the golden set on every prompt, retrieval, or model-version change; treat it as a regression suite, and do not tune against the same 60 flagged answers you started with - that overfits to the complaint rather than the task. Sample 20 live answers a week for human review, and watch for managed-model version changes in OCI Generative AI, which can shift behaviour without any change on your side.