Claims Intake: Regex vs. Managed Document AI
A regional insurer digitizes about 38,000 scanned pages a month: 11 recurring vendor layouts plus roughly 15% ad-hoc letters, with handwritten annotations on about 7% of pages. A pilot that sent every page to the OCI Document Understanding pretrained invoice model reached 62% exact-match accuracy on the six fields the claims system needs (claim number, policy number, incident date, provider, billed amount, total). Peak intake is 40 pages per minute, but 70% of pages arrive in two overnight batches. Compliance requires that national ID and phone numbers be redacted before any page is written to the data lake, and auditors want a per-field confidence trail. An engineer proposes dropping Document Understanding, running OCI Vision text detection on every page, and pulling fields with regexes.
1. Assess the Vision-plus-regex proposal and recommend an ingestion pipeline built from OCI AI services.
Read the worked answer
Why the proposal fails as the primary extractor. Text detection returns characters plus bounding boxes; it assigns no semantics. On a claim page with four dates, a regex cannot know which is the incident date, and multi-column tables interleave line items so a 'total' pattern will happily match a subtotal. Worse, its failures are silent: a pattern that stops matching after a vendor redesign produces a plausible wrong value, not an empty one. 11 layouts plus 15% ad-hoc letters is exactly the regime where hand-written patterns decay weekly.
Why it is still needed. Vision is the right tool for the handwriting share and for localization - you need bounding boxes to mask an ID on the page image.
Recommended pipeline
- Route pages with a Document Understanding document classification model (per-layout custom classes plus an 'ad-hoc' bucket).
- Extract with custom key-value models per high-volume layout; keep the pretrained invoice model for the layouts where it already scores well rather than retraining everything.
- Send low-confidence pages and the 'ad-hoc' bucket to a human review queue, with Vision text detection as assistance for handwriting.
- Run OCI Language PII detection over extracted text to find ID and phone spans, map them back to boxes, and mask the image before persisting.
The judgement: extraction is a field-semantics problem, redaction is a text-entity-plus-localization problem. One service does not cover both, and the cheap regex layer belongs only inside the review UI as an assist.
2. Only about 1,200 labeled pages are available and each label costs money. How should they be allocated, and what accuracy target is defensible?
Read the worked answer
Allocate by volume, but guarantee a floor per layout. Proportional-only sampling leaves rare layouts with 5-10 samples, which cannot validate a model. Take roughly 100+ pages for each of the top three layouts, 40-60 for each mid-volume layout, and accept a manual path for the tail.
Label only the six fields. Full transcription triples cost for data you do not consume.
Measure per field and per layout, never in aggregate. The pilot's 62% could be 95% on claim number and 25% on billed amount; averaging those hides that one field is production-ready and another is not. No single '90%' target is meaningful - set it per field from downstream cost. A wrong billed amount triggers rework; a missing provider is a phone call.
Close the loop. Any confidence threshold that routes pages to human review also generates free labels. Correct those reviews, retrain monthly, and track straight-through processing rate as the business KPI rather than raw accuracy.
Accept a ceiling. With 15% ad-hoc letters, a fully automated path is not the cheapest control; a well-designed manual queue for the tail is cheaper than chasing the last few points.
3. Where must redaction sit in the pipeline, and should processing be real-time or batch?
Read the worked answer
Redaction placement. Redacting only the extracted text field is insufficient - the ID is still legible on the page image, and that image is what lands in the data lake. Mask the identified regions on a derivative image, store that derivative as the working copy, and keep originals only in a restricted bucket with short retention and access logging. Run the redaction step before persistence, not as a later cleanup job; anything written to the lake unredacted is a reportable event.
Batch vs synchronous. 38,000 pages a month is about 1,260 a day, and 70% arrives in two overnight batches. That profile argues for asynchronous batch processing against Object Storage: it tolerates retries, absorbs the 40-pages-per-minute peak without provisioning for it all day, and decouples cost from idle capacity. Reserve synchronous per-page calls for the interactive review UI, where a human is waiting.
Audit trail. Persist per page: page id, routing class, model name and version, per-field value and confidence, whether a human corrected it. Auditors asked for a confidence trail, not a confidence average.