Contract Summarisation: Wrong Clause Numbers and a Fixed Inference Budget
A nine-person contracts team at a logistics firm reviews 250 supplier agreements a month, each 80-300 pages and full of cross-references such as 'see Schedule 4.2(b)'. They want a foundation model to produce a one-page risk summary with clause citations. A pilot using chunk-and-summarise produced fluent summaries, but in 12 of 40 sampled contracts the cited clause numbers were wrong, and liability caps were reported without the exceptions buried in a different schedule. Inference budget is fixed at about $2,000 per month. A 200k-token context model is available at roughly three times the per-token price of their current 32k model. Legal reviewers sign off on every summary, and their time is the firm's scarcest resource.
1. Diagnose the two failure modes and explain why further prompt tuning will not fix them.
Read the worked answer
Failure mode 1: invented clause numbers. The model was never given authoritative clause identifiers; it generated plausible-looking references from the prose. Prompting it to 'be accurate' does not supply the missing ground truth. Fix: extract clause IDs during ingestion as structured metadata, attach them to each chunk, and require the model to quote the clause text verbatim alongside the ID. A citation that cannot be matched to a retrieved span is discarded before the reviewer sees it.
Failure mode 2: exceptions lost across schedules. Chunk-and-summarise is a lossy map-reduce: each chunk is summarised in isolation, so a liability cap in the main body and its carve-outs in Schedule 7 never meet. No prompt wording repairs an architecture that physically separates them.
Both failures share a root cause - the pipeline treats a cross-referenced legal document as independent paragraphs. Tuning changes instructions, not the information available to the model. The fix is structural: chunk along document structure (clause, schedule), preserve the reference graph, and give the summary step access to both the cap and every schedule that qualifies it.
2. Using the cost figures, decide between single-pass long-context and structured chunking, and justify the choice on grounds other than price alone.
Read the worked answer
Rough arithmetic first. At ~500-600 tokens per page: 250 contracts x ~200 pages x ~550 tokens is about 27-28M input tokens per month.
- Long-context at, say, $6 per million input tokens: ~$165/month.
- 32k model at ~$2 per million: ~$55/month for a single pass, more once you add retrieval and re-ranking passes.
Both fit inside $2,000. Cost is not the binding constraint, so the decision must be made on accuracy and reviewer time.
Recommendation: hybrid, not either/or.
- Long-context global pass on each contract to enumerate structure - schedules, definitions, which clauses reference which - because this is exactly the cross-reference work chunking destroys. Output is a compact structured map, not prose.
- Targeted retrieval pass over the mapped sections to draft the risk summary, with verbatim quoted spans and clause IDs.
- Deterministic checks before human review: every citation must match a retrieved span; every liability-cap statement must be accompanied by the carve-out sections found in the map.
Justification outside cost: long-context single-pass recall degrades in the middle of very long inputs, and gives reviewers nothing to verify. Chunking alone gives verifiable spans but loses exceptions. The hybrid pays a modest premium for citations that survive review - which is the actual scarce resource.
3. Design the acceptance test that tells you whether the fix worked, given you can sample only 40 contracts.
Read the worked answer
Define the error taxonomy before looking at outputs. At minimum: (a) citation does not exist, (b) citation exists but does not support the claim, (c) material risk omitted (e.g., a cap exception), (d) summary correct but reviewer spent extra minutes verifying.
Measurement
- Paired design: use the same 40 contracts scored in the pilot, so you are comparing two versions on identical documents rather than on fresh noise.
- Citation precision: percentage of citations where the quoted span exists and supports the sentence. Audit exhaustively, not by sampling within a contract.
- Exception recall: for a checklist of known material risks per contract, was each captured?
- Reviewer minutes per contract: the operational metric that justifies the programme.
Statistical honesty. With 40 contracts and 12 prior citation errors, you can detect a large improvement (roughly, dropping to 2-3 errors) but not a subtle one. Two independent reviewers should score a subset to estimate disagreement, and results should be reported as counts with a confidence interval, not as 'fixed'. If the fix moves citation precision from 70% to 92%, say that and what remains.