Claims Extraction: Fine-Tune Versus Hosted LLM Under Residency Constraints
A health insurer must extract 14 fields - member ID, provider NPI, procedure codes, dates, billed amount, and nine others - from 60,000 claim documents per month. About 40% arrive as scanned images. A hosted LLM prompt achieves average field-level F1 of 0.88 at roughly $0.03 and six seconds per document, but the vendor endpoint processes data outside the insurer's region, which compliance prohibits for member data. A small open model fine-tuned on 3,000 labeled documents reaches 0.93 average F1 at $0.004 and 0.9 seconds, but the labels cover only six of the 14 fields well. The accuracy requirement is F1 ≥ 0.95 for ten fields and ≥ 0.90 for the remaining four. Training from scratch is on the table but has no realistic path within the 20-week deadline. A 12-person labeling team is available.
1. Recommend an architecture and justify it against cost, latency, residency, and accuracy.
Read the worked answer
The hosted endpoint is disqualified for member data by the residency constraint, so the realistic choice is the fine-tuned in-region model, with a larger in-region model or human review as fallback. Training from scratch is not credible in 20 weeks with 3,000 labels.
Cost and latency favor the small model strongly: 60,000 documents at $0.004 is about $240 per day versus about $1,800 for the hosted prompt, and 0.9 seconds versus six seconds matters at this volume. But the 0.93 average F1 hides the real problem: only six fields are well labeled, and the requirement is per-field, not average. A model can average 0.93 while several fields sit below 0.90.
I would deploy a confidence-routed pipeline: the fine-tuned model handles the six well-labeled fields and any high-confidence predictions; low-confidence fields and the eight weak fields route to a larger in-region model or a human queue. Scanned documents get OCR quality checks first. This meets residency, keeps unit cost low, and avoids claiming accuracy the labels cannot support.
2. How would you close the labeling gap for the eight poorly covered fields without exceeding the budget?
Read the worked answer
Start with stratified sampling by field presence, document type, and scan status, so the 12-person team labels the examples that actually contain the rare fields rather than random pages. Use LLM-assisted pre-labeling with human verification to cut annotation time, and reserve pure human labeling for the fields with the highest business or compliance risk.
Apply active learning: after an initial model, send the lowest-confidence and most disagreeing examples to annotation. Write field-specific guidelines and measure inter-annotator agreement; low agreement means the field definition is ambiguous and more labels will not fix it. For highly structured fields such as member ID or NPI, deterministic rules and checksums can supplement the model and reduce the labeling burden.
Set a per-field target, for example 400–600 verified examples for each weak field, and track F1 with confidence intervals. If a field cannot reach its threshold in time, keep it on human review rather than lowering the standard.
3. What risks do the scanned documents introduce, and how would you gate their release?
Read the worked answer
Scanned documents introduce OCR character errors, lost layout, rotated or low-quality images, and domain shift from the digital documents. Tables and handwritten annotations are especially fragile. The model may appear accurate on average while failing systematically on the 40% scanned segment, and errors in member ID or procedure codes can cause claim denials or compliance issues.
I would gate scanned processing behind a separate evaluation set drawn only from scanned claims, with per-field F1, character error rate, and confidence calibration reported separately. Add an OCR quality score and reject or route documents below a threshold rather than guessing. Confidence thresholds for scanned documents should be stricter, with a larger human-review band.
Release should be staged: shadow mode first, then a small percentage of scanned volume with full human verification, then scale only if per-field thresholds are met. Monitor OCR vendor changes, scan quality, and field-level error rates continuously, and keep a rollback path to human keying.