Fine-Tune or RAG for a Policy Q&A Bot That Keeps Going Stale
A compliance team wants a bot that answers employee questions about internal policy. There are 4,200 policy documents, updated weekly; roughly 3% of documents change each week and new ones appear monthly. A vendor demo showed a fine-tuned model answering in the house style. The team has 900 historical question/answer pairs, some outdated. Early tests show the fine-tuned model confidently cites a superseded reimbursement limit. Leadership wants the fine-tuned model because it "sounds more like us."
1. Diagnose the failure and recommend an architecture.
Read the worked answer
The symptom - confident citation of a superseded limit - is the classic signature of baking facts into weights. Fine-tuning teaches style and format well but encodes a snapshot of facts that cannot be selectively updated; retraining weekly on 4,200 docs is expensive and still lags. Worse, the 900 pairs include outdated answers, so the model is being taught stale facts as ground truth.
Recommendation: retrieval-augmented generation as the fact layer, fine-tuning (if at all) as the style layer. Index the 4,200 documents with metadata (effective date, supersedes, owner). At query time retrieve current-version passages and require the model to answer only from them, with citations. Weekly updates become a re-index of ~126 changed docs, not a retrain. If house style matters, fine-tune on style-only examples (or use a style prompt/few-shot), explicitly excluding factual Q&A pairs so no stale limits enter the weights.
2. What do you do with the 900 historical pairs?
Read the worked answer
They are contaminated as training data but valuable as an eval set and as style examples - with cleaning.
- Audit and date-stamp each pair against current policy. Pairs whose answers conflict with current documents are either dropped or relabeled with the correct current answer.
- Split: hold out a stratified eval set (include the tricky ones - changed policies, edge cases) to measure retrieval accuracy and answer faithfulness. Never train on the eval split.
- Style extraction: pairs that are factually current can inform tone/format, but prefer to strip the factual content or use them only as few-shot style exemplars rather than fine-tuning targets.
Guardrail: add a regression test that specifically asks about the superseded reimbursement limit and asserts the bot returns the current figure with a citation - the exact failure that triggered this.
3. How do you keep the system correct as policies change weekly?
Read the worked answer
Make freshness a pipeline property, not a model property:
- Versioned index with effective dates. Retrieval filters to documents effective as of today; superseded versions are retained for audit but never retrieved for current-answer queries.
- Change-triggered re-index on the ~3% weekly delta, with a smoke test that queries the changed docs and confirms the new text is retrievable.
- Citation-required answers so any wrong answer is traceable to a source; if no current source supports an answer, the bot says so rather than guessing.
- Monitoring: track retrieval-miss rate and "no supporting document" responses; a spike signals indexing lag.
This keeps the weekly cadence cheap and auditable, which a fine-tuned model cannot offer.