Choosing Grounding Over Fine-Tuning for a Policy Assistant
A compliance group wants an internal assistant that answers employee questions about travel and expense policy. The source material is 340 policy documents, revised quarterly, totaling about 1,800 pages. Answers must cite the exact section used, and any answer that cannot be supported by a document must be refused. A proposal suggests fine-tuning a foundation model on the full document set so it 'learns the policy'. A second proposal suggests indexing the documents and having the model generate answers from retrieved passages. The group has no machine learning engineers, only cloud administrators. Auditors will re-check 50 random answers per month against the documents.
1. Which proposal should be adopted, and why does it better satisfy the citation and refusal requirements?
Read the worked answer
Adopt the retrieval-based approach; reject fine-tuning on the documents as the primary mechanism.
Reasoning:
- Citations require traceability. If the model answers from retrieved passages, the passage itself is the citation and the auditor can compare answer to source directly. A fine-tuned model stores patterns in weights; it cannot point to the section it relied on, so citations would have to be reconstructed after the fact and could not be trusted.
- Refusal requires knowing when nothing supports the answer. Retrieval gives an explicit signal: if no passage clears a relevance threshold, the system refuses. A fine-tuned model tends to produce fluent policy-sounding text even when the corpus is silent, which is the opposite of the required behavior.
- Quarterly revision favors retrieval. Updating 340 documents in an index is an administrative task; re-running fine-tuning every quarter is an ML engineering task the team cannot staff.
- Auditing is easier. The 50-answer monthly sample can be traced to retrieved passages; with fine-tuning, auditors would be judging plausibility, not provenance.
Fine-tuning could still be considered later to shape tone or format, but not as the source of policy facts.
2. What failure modes should the team monitor once the assistant is live, and how would each show up in the monthly audit?
Read the worked answer
Monitor four failure modes, each with an audit signature:
- Retrieval miss. The right section exists but is not retrieved, so the assistant refuses a question it should answer. Audit signature: refusals on questions where a document demonstrably exists. Track refusal rate by topic.
- Retrieval of stale or superseded text. An older version of a policy is indexed alongside the current one. Audit signature: cited section conflicts with the current revision date. Enforce that only current documents are indexed and record document version with each chunk.
- Unsupported elaboration. The retrieved passage supports part of the answer but the model adds detail beyond it. Audit signature: auditor cannot find the extra claim in the cited section. This is the most dangerous mode because the answer looks cited.
- Citation mismatch. The answer is correct but cites the wrong section. Audit signature: content appears in a different section than the one named. Compare cited section text to answer text mechanically.
Operational steps: score each of the 50 sampled answers on supported/unsupported and correct-citation, report rates monthly, and treat unsupported elaboration as the top-priority defect because it undermines the refusal guarantee.