Contract Extraction: Choosing Between Prompting, Retrieval, Tuning, and Distillation

1. Schema enforcement (~hours, near-zero cost). Constrain output with function calling or a response schema rather than relying on prompt instructions. This alone should remove most of the 4% malformed-output failures. Fix the free win before anything expensive.
2. Prompt and input restructuring (~days). Chunk the contract by section headings instead of submitting 40 pages; extract field by field; require verbatim spans as evidence for each field. Most extraction failures are attention dilution or missing source text, not model capability.
3. Trim the few-shot set (~days). Twelve generic examples inflate cost and latency. Keep 3-4 examples chosen to disambiguate the fields that actually confuse the model (notice period versus renewal term, governing law versus venue). Re-measure; expect to retain most of the 74% gain at a fraction of the tokens.
4. Supervised tuning / adapter tuning (~weeks). With 5,000 annotated contracts this is viable and teaches schema adherence plus the extraction behaviour itself, typically beating prompt-only on a narrow, stable task and running cheaper at inference because the prompt shrinks.
5. Distillation to a smaller model (~weeks after 4). Generate teacher outputs on the 20,000 unlabelled contracts, filter by agreement with rule-based checks, train a small model for the month-end peak. Distillation compresses cost and latency; it does not add knowledge.

Ordering principle: exhaust changes that cost nothing to reverse before changes that require retraining and re-validation.

Retrieval belongs on the playbook, not on the contracts. The wiki contains interpretive guidance that changes quarterly - exactly the kind of knowledge that should be fetched at inference time so a policy update takes effect on the next query. Putting it in tuning data would bake in a version that silently expires, and the team would have no way to know when the model started applying last quarter's definition.

Contracts are the input, not a knowledge base. Each contract is processed once, in full or in sections, so there is little cross-document reuse to exploit. Retrieval across contracts becomes relevant only for portfolio questions ('which contracts have change-of-control clauses'), which is a different product.

Tuning belongs on behaviour: output format, section navigation, restraint on missing fields, span-extraction habits. Distillation belongs on the cost/latency axis after a teacher has demonstrated the quality ceiling. Keeping these three roles separate - retrieval for changing facts, tuning for stable behaviour, distillation for unit economics - prevents the common failure of retraining a model every time a policy document is edited.

• Holdout set: 500 contracts never used in tuning, stratified by contract type, counterparty region, language, and year, because acquisition-era templates differ materially from current ones.
• Per-field metrics, not aggregate. A 90% average can hide governing law at 55%. Report precision and recall per field with confidence intervals.
• Abstention testing: include contracts where a field is genuinely absent; the correct answer is 'not found'. Systems that guess on missing termination clauses are more dangerous than systems that flag for review.
• Adversarial slice: scanned or amended contracts, exhibits incorporated by reference, non-English agreements. These are where tuned models regress first.
• Confidence-gated routing: low-confidence fields go to a paralegal queue rather than straight to downstream systems; tune the threshold against the cost of an error versus the cost of review.
• Shadow deployment: run in parallel on live traffic for two weeks including a month-end peak, comparing against the current pipeline.
• Regression suite: freeze a golden set and re-run on every prompt, model, or tuning change; add each production failure to it. Watch for input drift when a new template appears.