Insurer's claim digitization: enrichment cost, indexer timeouts, and confidence-based review

The pipeline is paying for enrichment it does not need, repeatedly. $14,000/month and nightly full re-enrichment point at a structural problem, not a tuning problem: the skillset re-processes every document on every change and runs the most expensive steps unconditionally.

1. Route by cohort instead of running every skill on every file.

• 64% of documents have embedded text. Azure AI Search's built-in document extraction already parses those; running OCR over them is money spent to reproduce text you already have. Use a conditional skill gated on whether text was extracted.
• 6% handwritten + 30% scanned are the only documents that genuinely need Document Intelligence prebuilt-read. Send only those.
• Never send a document to the LLM skill unless a deterministic check failed (low confidence, schema validation failure, cross-field inconsistency). In most corpora that is a small fraction of pages.

2. Stop re-paying for the same page. Persist enriched output to an external store - a knowledge store (tables/objects) or a normalized JSON store - and index from that. Re-indexing, adding a field, or changing analyzers then costs search compute only. Bump a skillset version and reprocess incrementally rather than re-running OCR over 240,000 files.

3. Make enrichment incremental. Enable change detection and the indexer's deletion/update policies so only new or changed documents are enriched. Seed the corpus once; after that the daily job should touch a few hundred files.

4. Fix the timeout. The two-hour limit is per indexer run. Split the corpus across multiple indexers over different data sources (e.g., by year range), reduce batch size to keep individual executions short, and schedule them. For the one-time backfill of 240,000 files, drive it from an orchestrated batch pipeline rather than a single indexer run.

Judgement point. Do not start by downgrading the Document Intelligence tier or shrinking chunk sizes; both trade accuracy for cost on exactly the documents that already perform badly. Measure cost per cohort before and after routing - that is where the savings are legible.

Critique: 0.8 is a number, not a calibration. Model-reported confidence is not a probability of correctness. Document Intelligence field confidences come from a specific model and are typically optimistic on out-of-distribution inputs; 0.8 on a 2021 typed claim and 0.8 on a 1999 handwritten scan mean completely different things. Taking the rule literally gives you a review queue whose true error rate is unknown, and it may be both too large (wasting reviewers) and too permissive (letting bad fields through) at the same time.

Implementation.

1. Calibrate per cohort per field. Build a labeled holdout of roughly 500 documents per cohort (typed modern, scanned, handwritten) with the 14 fields checked by humans. For each field, compute actual accuracy in confidence bands (0.9–1.0, 0.8–0.9, 0.7–0.8, <0.7). Move the operational threshold per field so that the band you auto-accept meets the accuracy target compliance actually cares about. Tell compliance what you did and why - the 0.8 becomes a starting point, not a ceiling on your obligations.
2. Route on the minimum, but report the field. The rule triggers on any required field below threshold, so compute the minimum across required fields for the document, and send the reviewer straight to the offending field with the page image, the extracted value, the model's confidence, and the source cohort.
3. Cover fields that have no confidence. LLM-extracted fields do not emit a calibrated score. Don't invent one: use agreement between two extraction paths, schema/regex validation (dates, currency, policy-number checksums), and cross-field consistency (claim date after policy start) as the routing signal, and send any disagreement to review.
4. Close the loop. Track reviewer override rate by field and by confidence band. If reviewers change 30% of values in the 0.85–0.9 band, the threshold is wrong and should move. Feed corrected values back as labeled data.

Judgement point. Design the review queue so a reviewer can adjudicate a document in seconds; routing 40% of the corpus to review is a viable outcome only if each item is cheap to process.

Match the model to the cohort; one model will not serve 1998–2024.

• Typed 2015+ with stable layouts (preferred path): a custom extraction model - a template model where layouts repeat, a neural model where they vary - trained on a few dozen labeled examples per layout, or a prebuilt invoice/receipt model if the claim form is a standard invoice. These give per-field confidence and are cheapest per document at volume.
• Scanned image-only, typed content: prebuilt-read for OCR (text + layout + page coordinates) then a deterministic or custom field mapping. Do not send these to an LLM as the primary extractor.
• Handwritten adjuster notes: prebuilt-read handles handwriting but field-level accuracy will be materially lower; plan for mandatory review of fields sourced from these documents rather than trusting extraction.
• Legacy/variable layouts and cross-field reasoning (the fallback path): use an LLM extraction skill on the OCR output with a strict JSON schema and few-shot examples, only where cheaper paths failed. This is where the cost and hallucination risk live, so gate it.

Validate before the backfill.

1. Sample ~300 documents stratified across cohorts, vintages, and layouts; have humans label the 14 fields.
2. Run each candidate extractor on the same sample and compare field-level precision/recall, cost per document, and latency - not overall document accuracy, which hides the weak fields.
3. Decide per cohort from the data, then run a 2,000-document pilot end-to-end through indexing and review routing to validate throughput, cost per 1,000 documents, and queue volume before committing the remaining 238,000.

Judgement point. The pilot's projected review volume is the number that decides feasibility. If the calibrated thresholds imply 60,000 human reviews, the project cost is staffing, not compute - raise that before backfill, not after.

Indexing detail. Chunk with a split skill that respects page and section boundaries, and store page numbers plus the cohort tag in the index, so the assistant answering questions over claims can cite a page and so you can re-run a single cohort later without redoing the corpus.