Human Oversight and Confidence Handling in an Invoice Pipeline
An accounts payable team processes 9,000 invoices per month. A planned system extracts vendor, amount, due date, and purchase-order number from each invoice, then auto-posts it to the finance system. A pilot on 1,200 invoices extracted all four fields correctly on 1,080 invoices. On the remaining 120, errors clustered in scanned invoices with stamps or handwriting. The finance system has no undo for posted entries; corrections require a manual reversing entry that takes 20 minutes. Manual keying of one invoice takes 3 minutes. The team wants to auto-post everything the model is confident about and send the rest to a queue.
1. How should confidence be used to route invoices, and why is the current 'auto-post if confident' phrasing insufficient?
Read the worked answer
'Confident' is not a defined rule, and the pilot shows why the definition matters.
Facts to reason from:
- 1,080 of 1,200 (90%) were fully correct; 120 (10%) had at least one field error.
- Errors clustered in scanned invoices with stamps or handwriting, which is a detectable input characteristic.
- A wrong auto-post costs 20 minutes to reverse, versus 3 minutes to key manually. A bad auto-post is therefore worse than no automation for that invoice.
Routing design:
- Require per-field confidence, not a single document score. One wrong amount among three correct fields still corrupts the posting.
- Auto-post only when every required field clears its threshold and the invoice is not from the flagged input class (stamped or handwritten scans). Send everything else to human review.
- Send low-confidence extractions to review with the extracted values pre-filled, so the reviewer confirms or corrects rather than retyping. This preserves most of the 3-minute saving.
- Log the confidence score and the routing decision for every invoice so thresholds can be calibrated against actual error rates.
Economic check: a false auto-post costs 20 minutes, roughly seven times a manual key. The threshold should be set so the expected cost of errors is below the labor saved.
2. The team lead wants to auto-post all 1,080 'correct' pilot invoices to save time. What is wrong with evaluating the rule on the same invoices used to design it?
Read the worked answer
It is evaluating on the training data. The 1,080 figure was produced while looking at those same 1,200 invoices and choosing where to draw the confidence line. Any threshold tuned that way will look better than it performs on next month's 9,000 invoices, because the tuning absorbed the quirks of that sample.
Correct approach:
- Split the pilot invoices into a tuning set and a held-out set before setting thresholds. Choose the threshold on the tuning set, then report performance on the held-out set.
- Better, run a shadow period: process a few thousand live invoices with auto-posting disabled, record confidence and actual correctness, and set thresholds from that data.
- Watch for distribution shift. The error cluster was scanned invoices with stamps and handwriting; if the vendor mix changes, the error rate changes with it.
- Keep a monitored sample of auto-posted invoices checked by a human each month, because the finance system has no undo and errors otherwise surface only at reconciliation.
The lead's instinct to save time is reasonable; the flaw is the evidence base, not the goal.