Claims triage: fine-tuning versus retrieval at Northbridge Mutual
Northbridge Mutual, a mid-sized insurer, receives 62,000 motor claims a year. Its 14 claims handlers answer coverage questions by searching 900 policy wordings, some 120 pages long; wordings change monthly. A pilot assistant built by fine-tuning a 7-billion-parameter open model on 48,000 historical question-and-answer pairs gave fluent answers but invented two exclusions and could not cite the source clause. The proposed alternative is retrieval-augmented generation (RAG) over the current PDFs, returning answers with clause references. Constraints: the audit team requires traceability; the GPU budget is €3,000 per month; the model's context window is 32,000 tokens; average wording is 18,000 tokens but the largest is 90,000. Handlers will not accept answers that take more than 20 seconds.
1. Recommend fine-tuning, RAG, or a combination for this use case. Justify your recommendation against the audit, cost and freshness constraints.
Read the worked answer
RAG should be the primary architecture, with light fine-tuning only for style if needed. The knowledge that matters - exclusions, limits, wording changes - lives in documents that change monthly. Fine-tuning would bake a snapshot into weights, require retraining or refresher data each month, and still give no reliable citation. RAG retrieves current clauses at query time and can return clause references, satisfying audit. The cost profile also favours RAG: a monthly re-index is cheap compared with repeated fine-tuning on GPUs. The 32k context window means the 90k-token wording cannot be sent whole; the team must chunk by clause, embed, retrieve top passages, rerank, and generate an answer only from those passages. If a question spans multiple clauses, retrieve and cite each. A combination is defensible - fine-tune for house style and answer shape, RAG for facts - but the facts must come from retrieval.
2. Explain why the fine-tuned pilot hallucinated exclusions, and describe two controls that would reduce the risk in either architecture.
Read the worked answer
A generative model predicts likely tokens; it has no built-in truth check. In the pilot, the fine-tuned model learned patterns from 48,000 historical Q&A pairs, which may include outdated or incorrect answers, and it had no retrieval step forcing it to quote a current clause. When asked about an exclusion, it produced a plausible clause-like statement - a hallucination. Two controls: (1) grounded generation with mandatory citations and an abstain/refuse option when retrieved passages do not support an answer; (2) evaluation with a held-out adversarial set of coverage questions, including rare exclusions, plus human spot checks and confidence thresholds. Extractive answer modes or post-generation entailment checks against the cited clause add further assurance.
3. What does the 32,000-token context window imply for the 90,000-token wording, and how should the team handle it?
Read the worked answer
The context window is a hard limit on tokens the model can condition on at once. A 90,000-token wording cannot be passed whole into a 32,000-token window, and even the average 18,000-token wording leaves little room for the question, instructions and answer. The team should parse the PDF into clause-level chunks with metadata (policy, section, effective date), embed and index them, retrieve the most relevant chunks for each question, and pass only those to the model. Chunking must preserve clause boundaries and cross-references; a naive fixed-size split can separate an exclusion from its exception. Retrieval quality should be tested with questions whose answers span clauses. If a summary of a whole wording is needed, use hierarchical summarisation - summarise sections, then combine - and keep citations.