Supplier invoice extraction where line items do not sum to the total

Decision: prebuilt + custom neural + composed model, segmented by volume.

• Custom template models are the wrong tool here. They assume stable visual layout, which is why the PoC failed on rotated and skewed faxes. Do not retry template models with better preprocessing; the assumption itself is wrong for 14 varying layouts.
• Custom neural models tolerate layout and scan variation and need a realistic label set (roughly 5 minimum, ~15 recommended per model). Split by supplier group, not by document: one or two neural models covering the high-volume suppliers (the 6,000/month account and its peers) where labelled data pays back, plus the prebuilt invoice model for the long tail.
• Compose them into a single composed model so the pipeline has one endpoint and routes by supplier/ layout, simplifying downstream code.
• For the 40/month suppliers: labelling cannot be justified. Accept the prebuilt model plus human review, and quantify that review cost against a neural model's build cost before committing.

Trade-off: composed models add a routing failure mode - log which child model handled each document.

Do not ask the model to guarantee arithmetic; verify it outside the model.

• Extraction models predict fields independently. There is no mechanism that makes the predicted total agree with the predicted line items, so an 8% mismatch rate is expected and will not be fixed by more training data alone.
• Add a deterministic validation step: sum extracted line-item amounts (normalising currency, decimal separators, and locale formats), compare with the extracted total, and allow a small tolerance for rounding/tax presentation.
• Route mismatches, don't guess. Send them to a human review queue. Never auto-correct the total to the computed sum - the stated total may legitimately include freight, tax, or discount, and silent correction produces a wrong payment.
• Escalate extraction quality for mismatched docs: retry with higher-resolution or deskewed input, or flag low-confidence fields.
• Track a mismatch rate as a first-class KPI per supplier. A supplier whose mismatch rate is 30% signals a model or template gap, not a random error.

Also handle the handwritten totals: treat those fields' confidence separately; handwriting is the most likely source of the residual errors on the three fax suppliers.

Set an explicit confidence policy per field, not per document.

• Auto-post only when all critical fields (supplier, invoice number, date, total, line items) exceed a high confidence threshold and the arithmetic check passes and the supplier is a known, well-performing one.
• Everything else goes to review. The threshold is a business decision: compute expected cost of an erroneous auto-post versus cost of a review minute, and pick the threshold from that, not from what makes the model's numbers look good.
• Stratify thresholds by supplier. A 6,000/month supplier with 97% line-item accuracy should auto-post at a lower threshold than a 40/month fax supplier; one global threshold wastes review capacity or leaks errors.

Validation:

1. Build a labelled test set stratified by supplier and by scan quality - random splits hide the fax and rotated-scan failures that matter most.
2. Score field-level precision/recall (line items, total) plus end-to-end auto-post accuracy and review-queue rate.
3. Run a shadow period where the pipeline processes live documents without posting, and compare against the human-entered result.
4. Monitor after launch: review-queue rate, arithmetic mismatch rate, and per-supplier accuracy drift as layouts change.