Claims intake: extraction approach for a two-person, six-week deadline
A regional insurer receives about 38,000 claim documents a month: 70% clean digital PDFs, 30% scanned or photographed forms with handwriting and skew. Adjusters key 12 fields (claimant, policy number, incident date, ICD-10 code, billed amount, and so on) into a legacy system at roughly four minutes per document. Intake wants structured JSON landed in BigQuery. Staffing is two business analysts with no engineering support and no Python experience; the deadline is six weeks. Compliance requires that documents stay in us-central1, be processed only by Google-managed services under the existing BAA, and never be used to train a third-party foundation model. Volume is spiky: 400 documents on a normal day, 2,500 the Monday after a storm.
1. Recommend an extraction architecture for the 12 fields. Where, if anywhere, does a general-purpose multimodal model earn its place? Justify against the constraints rather than the benchmarks.
Read the worked answer
Start from the operating constraints, not model quality. Two non-engineers, six weeks, a fixed 12-field schema, and a contractual ban on third-party training point to Document AI custom extractor (or Form Parser with a custom schema): analysts label roughly 200-300 documents in the console, there is no code, the processor is region-pinned to us-central1, per-field confidence is returned, and document data is not used to improve a shared foundation model. The fixed schema also means per-page cost and latency are predictable, which matters at 38k pages/month.
A multimodal model (Gemini on Vertex AI) is not the primary extractor here. Per-page cost is higher, output variance on skewed handwriting is harder to bound, and the analysts cannot debug prompts the way they can review labeled pages. It does earn a place in two spots:
- Routing and triage - classify document type and page order before extraction; a cheap call on a cheap input.
- Confidence-gated fallback - when the extractor returns low confidence on a field, send only those pages to Gemini with a strict response schema, and log the result as a candidate, not as truth.
The failure mode to avoid is choosing the LLM first because the handwriting is hard, then discovering in week five that nobody can measure per-field accuracy or explain the cost curve.
2. There is no ML engineer and no test harness. How do you know the extractor is good enough to trust, and what stops quality from decaying after launch?
Read the worked answer
Hold out a golden set of ~400 documents, stratified so 30% come from the scanned/handwritten bucket and at least 50 come from storm-day batches. Label it once by adjusters, treat it as immutable, and report per-field precision and recall, not a single document-level score, because a wrong billed amount and a wrong middle name have very different downstream costs.
At inference, gate on confidence: fields below roughly 0.85 go to a human review queue. Expect 10-20% of fields initially; track that rate as the primary trend line. The business KPI is straight-through processing rate (documents posted with zero human touch), with a companion metric for error rate caught downstream - a rising STP rate with rising corrections is worse than flat STP.
To stop decay: every human correction is written back as a labeled example; retrain monthly on the accumulated corrections; re-run the golden set on every processor version before promotion, and freeze the version in the pipeline so an unattended auto-upgrade cannot change behavior on a Tuesday. Add a schema check on the output JSON so a renamed field fails loudly at ingestion instead of silently loading nulls into BigQuery.
3. Storm days spike volume 6x. How do you keep cost and latency sane without an engineer to build queueing infrastructure?
Read the worked answer
Separate interactive from bulk. Backlog and overnight batches go through Batch ProcessDocuments against a GCS input prefix, which is cheaper and tolerates latency; only documents that a human is actively working go through synchronous online processing. That alone absorbs the 2,500-document Monday without provisioning for peak.
Drive it from Cloud Storage event notifications into a Cloud Run job or a Workflow that chunks the batch, so no custom queue is needed - the analysts operate a console page and a bucket drop, not a scheduler. Set the batch job's concurrency and retry on the processor, and alert on documents older than N hours rather than on queue depth.
Cap the blast radius on the fallback path: on storm days the Gemini fallback will be invoked more often, so set a daily budget/quota on that path and route overflow to the human queue. Degrading to human review is acceptable; degrading to an unbounded LLM bill while missing the six-week deadline is not.