Routing 30,000 Contracts When the Biggest Model Is the Wrong Answer
A legal operations team must extract eight clause types from 30,000 supplier contracts averaging 45,000 tokens each, returning strict JSON for a downstream database. About 3% of contracts are escalated to a lawyer for interactive review, where a p95 response under 8 seconds matters; the rest run overnight. The team proposes routing everything to the largest, most expensive model, reasoning that accuracy is the only thing that matters. Their 200-document pilot showed 4% JSON schema violations, 11% missed clauses on scanned PDFs, and a projected monthly spend of $61,000 against an $18,000 budget. They have one week before the first batch of 4,000 contracts is due and no agreed definition of 'correct' extraction beyond the lawyer's intuition.
1. The team lead asks you to approve or reject the 'largest model for everything' proposal. What is your recommendation, and what would you need to measure before committing to the $18,000 budget?
Read the worked answer
Recommendation: reject as written, approve a routed design.
Three facts in the setup do the work:
- The dominant error (11% missed clauses) is concentrated on scanned PDFs. That is an input representation failure, not a reasoning failure; a larger model fed the same lossy text misses the same clauses. Money spent on model size buys nothing here.
- The workload is heterogeneous: 97% is overnight batch, 3% is interactive. Paying interactive-tier latency and price for the 97% is pure waste.
- $61k projected against an $18k budget is a 3.4x gap that no prompt tweak closes.
Design: route overnight extraction through the batch path (discounted, latency-tolerant, same model quality); reserve the low-latency path for the 3% escalation queue; gate every output through a schema validator plus a confidence check that re-routes failures to human review rather than silently passing them.
Measure before committing: a stratified sample of 300 contracts (at least 60 scans), double-annotated, reporting per-clause precision/recall and cost per 1,000 documents for each route. Commit budget only to the route that clears the threshold the lawyer signs off on in writing.
2. The 11% missed-clause rate is concentrated on scanned PDFs. A teammate proposes switching to a larger model to close the gap. Evaluate that proposal and describe what you would do first.
Read the worked answer
Switching models treats a data problem as a capability problem. Test the cheap hypothesis first: compare extracted token counts against expected tokens per page. If a 12-page contract yields 3,000 tokens of text, the extraction layer dropped tables and appendices and the model never saw the clause.
Diagnostic sequence:
- Take 20 known-missed scans and inspect the text actually sent. Clause present but missed? Then it is reasoning or instruction quality. Absent or garbled? Then it is OCR/parsing.
- If absent, add an OCR or vision path for image-heavy pages; keep the text path for digital-native PDFs. This is a per-document routing decision, not a model upgrade.
- If present but missed, the usual causes are clause position (buried mid-document) and vague clause definitions. Fix with explicit per-clause definitions and a per-clause extraction instruction before touching model size.
Only once the text layer is clean is a model comparison meaningful. Run both models on the same cleaned input; if the difference is smaller than inter-annotator disagreement on the sample, keep the cheaper one. The 11% is a signal about the pipeline, and the fix is upstream of the model.
3. The 4,000-contract batch is due in one week. Sketch a delivery plan with explicit checkpoints and state what you would do if a checkpoint fails.
Read the worked answer
Five working days, 4,000 contracts.
- Day 1: freeze prompt and schema; agree acceptance thresholds in writing (e.g. ≥95% clause recall on a 100-doc sample, 100% schema validity, ≤2% escalated to humans). No prompt edits after freeze except blocking defects.
- Day 1–2: run a 400-document (10%) pilot through the chosen route. Send 100 of them through both the batch and interactive paths to confirm batch quality parity - do not assume it.
- Day 2: human-verify the 400 against thresholds. This is the go/no-go point.
- Day 3–4: process the remaining 3,600 in batch chunks with automatic schema validation and re-queue for failures. Track failure rate hourly; halt if it exceeds 5%.
- Day 5: deliver accepted output, route the escalation queue (~120 contracts) to lawyers interactively, and produce a defect log that becomes next month's regression set.
If day-2 recall is below threshold: do not scale. Deliver the 400 with disclosed limitations and renegotiate the deadline. A late 400 is recoverable; 4,000 bad extractions in a contract database is not.