Stale rate answers in a credit-union knowledge assistant
Northline Credit Union runs a watsonx Assistant knowledge-base assistant for 14,000 members. The corpus contains 1,200 policy PDFs and 300 FAQ articles in a single collection. Retrieval returns the top eight chunks; chunk size is 1,500 tokens with no overlap, and the PDFs include rate tables. The generative answer step is set to temperature 0.7 and is allowed to answer even when search results are weak. In a sample of 300 logged questions, 22% of answers cite a document that does not contain the stated figure. A member asking about the current auto-loan APR was told 7.9%, a rate from a 2021 marketing brochure; the current rate is 6.4%. The team wants to switch to a larger LLM and rewrite the prompt. The compliance lead wants evidence that answers are grounded before the next audit. The assistant has Draft and Live environments, and the next change window is in five days.
1. Diagnose whether the primary failure is retrieval, generation, or both. Cite the evidence you would use and explain what you would check first.
Read the worked answer
The primary failure is retrieval/grounding configuration, not model capacity. The strongest evidence is the citation mismatch: 22% of answers cite a document that does not contain the figure. A generative model can hallucinate, but a specific stale rate such as 7.9% from a 2021 brochure usually means the retriever surfaced that brochure and the generator copied it. Check the failing 30 questions by logging retrieved chunk IDs and whether the gold passage was in the top eight. If recall@8 is low, fix retrieval. If the gold passage is present but the answer is wrong, inspect the prompt and answer settings. Also check table extraction: 1,500-token chunks with no overlap often split rate-table headers from values, so the model sees 7.9 without 2021 or auto loan. Temperature 0.7 is secondary; it increases paraphrase variance but rarely invents a precise stale number. The setting that allows an answer when search results are weak is a major risk because it forces an answer when no current passage exists. A larger LLM would make the wrong answer more fluent, not more grounded.
2. Propose a remediation plan ordered by risk reduction per unit of effort, including specific settings and content changes.
Read the worked answer
Order by risk reduction. First, content hygiene and metadata: inventory all 1,500 documents, add effective_date, product_line, doc_type, and superseded_by; archive superseded brochures and filter retrieval to current, product-matched documents. This alone removes the 7.9% source. Second, table-aware ingestion: convert rate tables to row-level records or HTML tables with headers; chunk at 300-500 tokens with 10-15% overlap so a rate and its effective date stay together. Third, retrieval: top-k 5 with reranking, query rewriting for product acronyms, and metadata filters for member segment. Fourth, grounding: require citations; instruct the model to answer only from retrieved passages and to abstain when no passage supports the figure; set an abstention or confidence threshold; lower temperature to 0-0.2. Only then test a different model. The largest risk reduction per effort is removing stale documents and adding date filters because it eliminates the exact failure mode. Prompt rewriting first would be cosmetic. Keep a rollback copy of the old index.
3. Define an evaluation that would convince compliance the assistant is grounded, including metrics, test-set construction, and pass/fail thresholds.
Read the worked answer
Build a 150-200 question golden set from real logs plus compliance cases, stratified by product (auto, mortgage, credit card), answer type (rate table, narrative policy, eligibility), and answerability (answerable versus out-of-scope). For each, record the gold answer, authoritative document ID, and supporting span. Have two reviewers label independently and resolve disagreements. Metrics: retrieval recall@5 (gold span retrieved), citation precision (cited document contains the answer), numeric accuracy (exact rate or date), semantic answer accuracy, abstention on unanswerable questions, and stale-document citation rate. Pass/fail: numeric accuracy at least 98% for rate questions, overall answer accuracy at least 95%, citation precision at least 95%, abstention at least 90% on unanswerable questions, and zero stale citations on current-rate questions. Run first in Draft, then a 10% canary in Live with human review of flagged answers. Do not rely on an LLM judge until it agrees with human labels on 50 cases. Report per-product slices, not just an average, because auto-loan tables may fail differently from mortgage narratives. Include a regression run before every publish.