Evaluating a Summarization Feature Before Release
A legal operations team wants a feature that summarizes 40-page vendor contracts into a one-page brief covering parties, term, termination rights, liability caps, and governing law. A prototype produces fluent briefs, but in a review of 60 contracts, 9 briefs omitted the liability cap entirely and 4 stated a termination notice period that did not match the contract. The team lead argues the prototype is '90% good' and should ship with a disclaimer. The contracts are high value; a missed liability cap has caused disputes before. Reviewers currently read every contract in full, taking 90 minutes each.
1. Is '90% good' an adequate framing? Evaluate the prototype against the actual use.
Read the worked answer
No. A single aggregate score hides the failures that matter.
Breakdown of the 60-contract review:
- 9 of 60 (15%) omitted the liability cap. That is a required field, not a nice-to-have. An omission is silent: the brief looks complete, so a reviewer who trusts it will not know to look.
- 4 of 60 (about 7%) stated a wrong notice period. This is worse than omission because it is confidently wrong and invites reliance.
- The remaining briefs may be fine, but the two defect classes above are precisely the ones that cause disputes.
Judgement: '90% good' averages a harmless formatting quirk with a missed liability cap. The correct framing is per-field recall on required fields and factual accuracy on extracted values. A feature that silently drops a required field at 15% is not shippable as a substitute for reading, regardless of fluency.
Also note the baseline: reviewers spend 90 minutes per contract. The feature only creates value if it reduces that time without increasing missed-cap risk.
2. Propose a measurement plan and a release decision rule.
Read the worked answer
Measurement plan:
- Build a labeled evaluation set of at least 100 contracts, each annotated by a lawyer with the five required fields and their exact values. Include contracts with unusual liability clauses and short notice periods, since those are where extraction fails.
- Score per field, not overall: presence recall (was the field found when present?), value accuracy (does the stated value match?), and hallucination rate (fields stated that are not in the contract).
- Score the brief as a whole for any statement not supported by the contract, since fluent unsupported text is the main risk.
- Run the evaluation on a held-out set after any prompt or model change; do not tune and report on the same contracts.
Release decision rule:
- Presence recall on liability cap and termination rights must be at or above 98%, with zero confident wrong values tolerated above 1%.
- Any contract where the model cannot find a required field must say so explicitly rather than omitting it silently.
- Ship first as a draft that reviewers must confirm field by field, and only remove the full read for contract types that clear the bar across two consecutive evaluation rounds.
A disclaimer does not substitute for these thresholds, because the harm is missed information, not offense.