Invoice extraction at 40,000 calls a day: few-shot, schema, or fine-tune?
A shared-services team extracts 12 fields (vendor, invoice number, dates, currency amounts, PO number, tax, payment terms) from 40,000 invoices per day. The current design is a zero-shot prompt on Granite-13b-instruct with a 1,900-token instruction block, JSON parsed by regex with a retry-on-parse-failure fallback. Field-level exact match is 71%. The dominant failures are date normalization, currency symbols, and amounts split across line items; semantic misreading of vendor names is rare. Adding 8 few-shot examples raises exact match to 89% but grows the prompt to roughly 3,100 tokens and pushes p95 latency from 2.1s to 4.1s. The platform budget allows 25 million input tokens per day for this workload. Straight-through processing requires 95% field-level accuracy, and 30% of invoices are scanned PDFs with OCR noise. The team proposes fine-tuning Granite on 2,000 labeled invoices and dropping the few-shot examples.
1. Should the team fine-tune as proposed? Show how you would decide, including the cost arithmetic.
Read the worked answer
Start from the error taxonomy, not the proposed solution. The failures are overwhelmingly normalization and formatting (dates, currency, line-item amounts), not domain semantics. Few-shot examples mostly teach output shape and formatting conventions, which is why they moved accuracy from 71% to 89% without teaching the model anything new about invoices.
Do the cost arithmetic before anything else. At 3,100 tokens × 40,000 calls, the few-shot design needs about 124M input tokens/day, roughly 5× the 25M budget. Even a trimmed 800-token prompt needs 32M, still over budget. So the current direction is not viable regardless of accuracy.
Fine-tuning can help, but 2,000 examples across 12 fields is thin for rare formats, and it does not fix deterministic problems. A better sequence:
- Enforce a JSON schema with constrained decoding, so malformed output and wrong types disappear structurally.
- Normalize dates and amounts in post-processing with validation rules (ISO 8601, currency code table, sum of line items equals total).
- Shrink the prompt: move invariant instructions into a compact system prompt, keep 2–3 examples covering the formats that actually fail.
- Route: small model for clean digital PDFs, escalate OCR-noisy documents to the larger model or human review.
Fine-tune only if, after these steps, semantic confusion remains in the error sample. Evaluate on a held-out set split by vendor and by scanner, not randomly by invoice.
2. Design the accuracy measurement and low-confidence handling needed to reach 95% straight-through processing.
Read the worked answer
Field-level exact match is the right headline metric, but it hides the difference between a wrong date and a wrong amount. Weight by business harm: amounts, PO number, and tax are payment-critical; payment terms and vendor name affect routing. Report per-field precision, recall, and a critical-field accuracy figure separately.
Measure at three levels:
- Parse validity: percentage of responses that validate against the JSON schema without repair.
- Field accuracy: exact match after normalization, compared against a human-labeled golden set of at least 500 invoices stratified by vendor, format, and scan quality.
- Straight-through rate: percentage of invoices that pass all validations with no human touch, plus the human-correction rate on the remainder.
Low-confidence handling: derive a confidence signal per field (token log-probabilities, agreement across two sampled decodes, or a validator outcome), set per-field thresholds rather than one global threshold, and route below-threshold fields to a review queue with the source crop highlighted. Add hard rules that override the model: line items must sum to the total, dates must parse, currency must be in the allowed list. Any hard-rule violation is auto-escalated.
Track a weekly sample of auto-approved invoices reviewed by humans; this estimates the silent error rate and tells you whether the 95% target is real or an artifact of the golden set.
3. A stakeholder asks why not simply use the largest available foundation model and skip all this engineering. How do you answer?
Read the worked answer
A larger model improves semantic understanding but does not change the economics or the failure modes here. At 40,000 calls/day, a model that costs 3–5× more per token blows the 25M-token budget even with a short prompt, and latency at p95 would worsen queueing during month-end peaks.
The dominant errors are deterministic: date formats, currency symbols, amounts split across rows. A larger model will still emit 03/04/24 ambiguously or drop a decimal unless you constrain the schema and validate. Paying for scale to solve a formatting problem is the expensive path.
The defensible architecture is tiered: deterministic parsing and validation first, a compact model for the majority of clean documents, and escalation to a larger model only for the residual cases where semantics genuinely matter (multi-vendor consolidated invoices, unusual layouts, OCR-degraded scans). That keeps cost bounded, keeps latency predictable, and concentrates human review where it adds value.
Frame it as an accuracy-per-dollar decision with an explicit escalation budget: for example, allow 8% of volume to hit the large model, compute that cost, and show it fits. Stakeholders accept a tiered design when the escalation path is visible and the critical-field error rate is the metric being optimized.