A 14-category clause classifier with noisy labels and three weeks
A legal operations team wants to classify contract clauses into 14 categories and extract 5 attributes (party, effective date, term length, termination notice, governing law). They have 4,800 labeled clauses produced by three reviewers whose pairwise agreement is 81% overall but only 62% on two categories that are frequently confused. Class distribution is imbalanced: the rarest category has 90 examples, the largest has 1,100. The team has 2 GPUs, three weeks, and a requirement of macro-F1 ≥ 0.85 on held-out contracts, plus an explainability requirement from internal audit: a reviewer must be able to see why a clause was classified as it was. Options on the table are few-shot prompting of Granite-13b, LoRA fine-tuning of Granite-8b, or training a small encoder classifier alongside a generative extractor. Clauses come from 900 contracts, roughly 5.3 per contract.
1. Which approach would you choose, and how would you split the data?
Read the worked answer
Run a fast few-shot baseline first - a day, not a week - to see where Granite-13b lands on the two confusing categories. If macro-F1 is already near 0.80, prompting plus a schema may be enough and the project becomes an evaluation problem rather than a training problem. Assume it is not.
For 4,800 examples and 14 categories, a fine-tuned encoder classifier is usually the strongest and cheapest classifier, and it gives you attention or token-importance evidence that audit can inspect. LoRA on Granite-8b is more flexible but will not beat a well-tuned encoder on a 14-way classification with this data volume, and it costs more to serve. My choice: encoder classifier for the categories, generative model with constrained decoding for the 5 attributes, because date and term extraction benefit from language understanding and schema enforcement.
Splitting is the trap. Split by contract, not by clause: clauses from the same contract share boilerplate and will leak across train and test, inflating macro-F1 by several points. Hold out about 15% of contracts stratified by contract type and by time (prefer the most recent contracts in the test set), keep a dev split for threshold tuning, and never tune on the test set.
2. How do you handle the 62% agreement on two categories and the class imbalance?
Read the worked answer
The two low-agreement categories are a labeling problem, not a model problem. Spend two days on adjudication: have a senior reviewer resolve a sample of disagreements, write a decision rule with examples for each category boundary, and re-label the disputed cases. If the boundary is genuinely subjective, consider merging the two categories or adding an explicit ambiguous label; forcing a model to reproduce inconsistent labels caps achievable F1 and makes the 0.85 target meaningless. Weight or exclude cases that remain disputed, and report inter-annotator agreement in the model documentation.
For imbalance: use class-weighted loss or a balanced sampler during training rather than duplicating text, and evaluate with macro-F1 so the 90-example class counts as much as the 1,100-example class. For the rarest category, collect or actively select more examples - 90 is enough to evaluate, barely enough to learn. Per-class decision thresholds tuned on the dev set will buy more than architecture changes.
Finally, measure calibration, not just F1. Audit needs to know when the model is unsure; a classifier that is 0.99 confident on the two confusing categories is worse than one with lower accuracy but honest probabilities, because the latter routes to human review correctly.
3. What evaluation and guardrails would you require before audit signs off on the attribute extraction?
Read the worked answer
Evaluate attributes separately from classification, with field-level precision and recall on held-out contracts: party, effective date, term length, termination notice, governing law. Report exact-match and normalized-match (dates in ISO form, notice periods converted to days) so a formatting difference is not counted as a reasoning error.
Guardrails for extraction:
- Span grounding. Every extracted value must be returned with the source span it came from. If the model cannot ground the value in the clause text, the field is marked missing rather than guessed.
- Schema and type validation. Dates must parse, notice periods must be positive numbers, governing law must come from an allowed jurisdiction list. Invalid values go to review.
- Abstention is a valid output. Train and prompt for
not_found; measure abstention precision so the model is not rewarded for guessing. - Confidence routing. Low-confidence fields go to a human queue; track the review rate as a cost metric, not just an accuracy metric.
For auditability, keep a factsheet with the training data provenance, the adjudication decisions, per-class metrics, known confusions, and the evaluation dataset version. Explainability outputs should show the tokens that drove the classification and the span behind each extracted value, so a reviewer can verify in seconds.