Tailwind Logistics: Fine-Tune, Extend, or Stop the Invoice Automation
Tailwind Logistics processes 40,000 supplier invoices a month, 15% of them handwritten. A four-week proof of concept using an Azure OpenAI model returned 82% field-level accuracy. Finance requires 97% accuracy on total amount and supplier tax ID before any invoice can skip manual keying. Manual processing costs $0.42 per invoice. The team's Azure OpenAI resource sits in East US, but the data protection officer requires storage and processing in West Europe, and invoices contain bank details. The sponsor needs a decision in two weeks on whether to fine-tune, extend the pilot, or stop, and has $18,000 per month of AI budget.
1. Should the team fine-tune, extend the pilot, or stop? Build the argument the sponsor should hear.
Read the worked answer
The 82% figure is an average across all fields and all document types, which makes it almost useless for the decision. Resegment it before deciding anything: printed versus handwritten, and critical fields versus line-item fields. Total amount and tax ID on printed invoices are usually well above the average; handwriting and free-text line items drag it down.
That segmentation changes the architecture. Use Azure AI Document Intelligence for layout and OCR, and reserve the language model for normalisation and exception reasoning rather than raw extraction. Add deterministic validation: line items must sum to the total, the tax ID must pass a checksum and match the supplier master, and the bank details must match the last approved record for that supplier. Route anything below a per-field confidence threshold to a human queue, and treat the 15% handwritten stream as human-assisted from day one.
Fine-tuning is the wrong first lever: it teaches format conventions, not new layouts, and supplier invoices change constantly. Expand the pilot by four weeks with the segmented design and a target of 97% on the two critical fields with zero silent auto-post errors. Predefine the stop condition: if the critical-field target is not met, keep human-in-the-loop and drop the straight-through-processing ambition instead of buying more model.
2. How do you meet the West Europe requirement, and what has to be re-tested because of it?
Read the worked answer
Region is a hard constraint, not a deployment detail. Create the Azure OpenAI and Document Intelligence resources in a West Europe region, and verify model availability in that region before promising dates; not every model and version is offered everywhere, and that gap is the most common cause of teams quietly inferring in another geography.
Re-test everything that depends on the model or the region: accuracy by field on the same 500-invoice sample, because a different model version changes the confusion matrix; latency, because the call path changes; and the confidence calibration, because the thresholds that routed to human review were tuned on the previous model. Re-run cost per invoice against the new resource's pricing.
Confirm the data plane explicitly: invoice images, extracted text, and logs all remain in-region, including logs retained by any evaluation tooling and any index used for the supplier master. Bank details push this into a higher classification, so document retention and access roles need sign-off, not just a region setting.
If a required model is unavailable in-region, the answer is a different model or a documented approval - never silent cross-region inference, which the DPO would treat as a reportable deviation.
3. Given manual cost of $0.42 per invoice, what does the 97% target have to be worth? Challenge the budget.
Read the worked answer
Run the arithmetic before conceding to the $18,000 budget. 40,000 invoices at $0.42 is $16,800 per month of total manual cost. An AI programme costing $18,000 a month is more expensive than the entire process it replaces, before counting exception handling, integration, and human review of low-confidence documents. The business case cannot rest on labour savings alone, and the sponsor needs to hear that in week one, not in month six.
Rebuild the case on three levers. First, cost per document: only send low-confidence documents to the model, use batch processing where SLAs allow, and let Document Intelligence handle the printed majority so model tokens are spent on exceptions. Second, working capital: earlier capture can unlock early-payment discounts and avoid late-payment interest - quantify both with finance. Third, avoided error cost: duplicate payments and mis-keyed bank details have a real historical value; ask for the last twelve months of adjustments.
Then set the target against that number. If only 85% of volume can skip keying once handwriting is excluded, the saving is roughly $14,300 against a cost that must fall well below it. Set the stop rule accordingly: proceed only if projected monthly AI cost is under half the validated benefit, otherwise narrow scope to printed invoices from the top 20 suppliers and revisit.