Claims document extraction with a two-person analyst team
A regional insurer processes about 12,000 claim documents a month: faxed scans, emailed PDFs, and two portal-generated layouts. Adjusters must pull nine fields (claimant name, policy number, date of loss, ICD-10 code, billed amount, provider NPI, and three free-text notes) into a claims system. Two contractors currently key them by hand. The team owns two business analysts and one integration developer; there is no ML engineer and the deadline is six weeks. Leadership also wants adjusters to ask natural-language questions such as 'what is our policy on cosmetic procedures after a car accident' and get answers drawn from 4,000 internal policy pages, at roughly 300 questions a day. Per-page processing cost is under scrutiny from finance.
1. Recommend an extraction approach for the nine fields and justify it against at least one alternative.
Read the worked answer
The decision hinges on three properties of this workload, not on which product is newest.
- The schema is fixed and high-volume. Nine known fields at 12,000 pages/month is a classic structured-extraction problem. A Document AI custom extractor gives you per-field confidence scores, normalized entity types (dates, currency, IDs), and predictable per-page pricing - all of which the analysts need for routing and for a cost forecast finance will accept.
- Layouts are not uniform. Two portal layouts are stable; faxes vary. A single custom extractor trained across all layouts usually underperforms one trained on the stable layouts plus a fallback path.
- The alternative is a Gemini prompt over the page image. That needs no labeled training set and handles messy free-text notes well, but confidence is not calibrated per field, output can vary run to run, and cost per page is harder to pin down at scale.
Pragmatic answer: a hybrid. Document AI custom extractor for the two high-volume portal layouts (label ~100-150 docs per layout), Gemini for the low-volume, high-variance fax tail and for the three free-text notes. Route by a cheap layout classifier rather than trying to force one processor to do everything, and re-measure after four weeks before locking the split.
2. How should the policy Q&A feature be built so adjusters can trust it, without an ML engineer on staff?
Read the worked answer
Use a grounded retrieval setup rather than a bare chat prompt: load the 4,000 policy pages into Vertex AI Search, then expose them through an Agent Builder / conversational agent whose answers are grounded in retrieved passages with citations back to the source page.
Reasoning: the failure mode that destroys trust here is a confident answer that cites no policy. Grounding plus citations makes each answer checkable in seconds, and the agent can be configured to decline when retrieval scores are low instead of improvising. 300 questions a day is trivial load, so latency and cost are not the constraint - correctness is.
Two governance decisions matter as much as the tech:
- Scope it as advisory. The tool answers 'what does policy say', not 'should we pay this claim'. Say so in the UI.
- Own the corpus. Policy pages drift. Assign one analyst to re-index on a schedule and to review a sample of 20 answers a week; log questions that got declined, because those are your content gaps.
Avoid fine-tuning on policy text: it bakes in stale content and removes the citation trail.
3. Where should humans stay in the loop, and how would you decide the threshold?
Read the worked answer
Set the threshold from a labeled holdout set, not from intuition. Take ~300 documents the extractor has never seen, compute per-field precision/recall at several confidence cutoffs, and pick the cutoff where the error rate on auto-accepted fields is below what the claims team says is tolerable (often 1-2% for amounts, near-zero for policy number because downstream matching fails loudly).
Route anything below the cutoff to the existing contractor queue - the contractors become exception handlers rather than keyers, which is also the change-management story you tell them.
Two things to watch after launch:
- Confidence distribution drift. If the share of documents below threshold creeps up, a new fax format or scanner has appeared. Alert on that percentage, not only on accuracy.
- Silent failure on free text. The three notes fields will be the weakest; sample them weekly even when confidence is high.
Revisit the threshold quarterly, and only after the extractor has been retrained, since calibration shifts with the model.