Tailspin Legal: contract extraction and matter-scoped search
Tailspin Legal holds 1.2 million pages of supplier contracts scanned between 1998 and 2024 - mixed typewritten and faxed, with inconsistent quality. They process 80,000 invoices a year across four known layouts plus a long tail of new suppliers. Commercial needs 14 fields per contract, including party, effective date, renewal term, termination notice, liability cap and governing law. Compliance requires 97% field accuracy overall with a fully auditable human review trail for anything below threshold; their current keyword-and-regex pipeline scores 68% on liability cap. Legal also wants natural-language search across the contracts and 400,000 related emails, respecting matter-level permissions, with answers citing the source page.
1. Choose the extraction approach for invoices and for contracts, and justify why the regex pipeline fails on liability cap.
Read the worked answer
Invoices: composed model, not one model
- Try the prebuilt invoice model first - standard fields with no training and strong accuracy. Train only on the delta.
- The four known layouts are fixed-structure: a custom template model needs about five labelled samples per layout and generally beats neural on fixed structure.
- The long tail of new suppliers is varied: a custom neural model handles layout variation without per-supplier labelling.
- Compose template + neural + prebuilt into one model ID, and put a custom classification model in front to route each PDF, so the pipeline does not need manual routing or filename conventions.
Contracts: neural base, with an OCR quality gate 1.2M pages of 1998-2024 scans with 14 fields and unpredictable layout rules out template models except for any genuinely boilerplate families. Run prebuilt-read first and flag pages whose OCR confidence is low: faxed pages that defeat OCR will defeat extraction, and that must be visible as a distinct failure rather than a wrong value.
Why regex fails on liability cap The field is prose, not a slot: 'shall not exceed 2× fees paid in the preceding 12 months', 'capped at £250,000', 'limited to the greater of…'. A regex encodes one phrasing and scores 68% because it matches the template it was written against. Use a model that extracts the value and the source span, so a reviewer can verify the number against the clause - which is also what the audit trail needs.
Rule to state: standard document type → prebuilt; fixed layout, high volume, exact fields → template; variable layout → neural; compose them rather than choosing one.
2. How do you reach and defend 97% field accuracy with an auditable review trail?
Read the worked answer
97% is not one number. It is the precision on auto-accepted fields, plus the correction rate on reviewed fields, measured per slice. A single aggregate figure can hide faxed 1999 contracts at 70%.
Routing
- Use per-field confidence from the model; route any field below a tuned threshold to a review queue. Tune the threshold so the auto-accepted set clears 97%, not so that the queue is small.
- Route whole documents when page OCR quality is low or the document classifier is unsure - those are systematically worse.
- Never mark a below-threshold field as final without review just to hit throughput.
Measuring
- Stratify the evaluation set by layout, supplier, year and scan quality. Report per slice and set the acceptance gate on the worst slice.
- Track precision on auto-accepted fields continuously (sample-audited) as well as reviewer corrections; the two drift in opposite directions.
Auditable trail Persist, append-only, per extracted field: raw model output, confidence, model version, document/page reference, reviewer identity, original and corrected value, timestamp. This satisfies compliance and doubles as training data - corrections become labelled examples.
Where 97% may be unreachable (free-text liability cap in poor scans), negotiate the target: 97% on structured fields, mandatory human review for the prose fields, or re-scan. Promising 97% uniformly is how the project fails its audit.
3. Design the natural-language search over contracts and emails, including permissions and page-level citation.
Read the worked answer
Index design
- Azure AI Search with one indexer per source. Emails have no first-party indexer, so export to a supported source (Blob/SharePoint) or push via the REST API; contracts land as PDFs in Blob.
- Integrated vectorization: document cracking → split into chunks (preserve page number and heading) → embedding skill → vector field. Also project the 14 extracted contract fields as filterable/searchable fields so 'contracts with a 90-day termination notice' is a filter, not a hope.
- Hybrid retrieval (vector + keyword + semantic ranker) - clause identifiers and party names need exact matching while the question is natural language.
Page-level citation Document Intelligence layout output gives page numbers; carry page and document ID in every chunk, return them with the result, and require the answer to cite chunk IDs. Validate that cited IDs were in the retrieved set - otherwise you have plausible-looking citations that point at nothing.
Permissions (the part that usually goes wrong)
- Store the matter's authorised group IDs on every chunk as a filterable collection field, inherited by emails from their matter.
- At query time, resolve the user's group membership from Entra ID and add a mandatory security filter: the user's groups must intersect the chunk's ACL. Security trimming must be server-side; never filter in the UI.
- Never cache or log unfiltered result sets; a shared semantic cache can leak documents across matters.
Freshness and scale: incremental indexer runs plus push for new extractions; test query latency with the filter applied, since the filter changes the candidate set.
Validate with a red-team test: a user with no matter access queries for a known clause and must receive zero results.