Truncation at 512 Tokens Hides Late Clauses in Chinese Contract Review
A legal-tech team fine-tunes a Chinese BERT-base model for multi-label clause classification on 120,000 contracts. Documents average 3,000 Chinese characters, and 8% exceed 8,000. The tokenizer produces about 1.3 tokens per character. The team truncates every document to the first 512 tokens, trains with binary cross-entropy, and reports micro-F1 0.86 on a random held-out split. A pilot with 300 real contracts shows that clauses appearing after character 512 are found with recall 0.31, while early clauses reach 0.88. Latency budget is 200 ms per document on a single Ascend 310 inference card; a 512-token BERT-base forward pass takes about 35 ms. The product owner wants to switch to a long-context model immediately. The team has not measured how often late clauses occur or their business cost.
1. Quantify the problem. What would you measure before changing the model?
Read the worked answer
The first correction is units. At 1.3 tokens per character, 512 tokens cover roughly 394 Chinese characters, not 512. Truncation is therefore more aggressive than the team believes, and the reported micro-F1 describes only the first several hundred characters.
Before changing architecture, measure:
- Label position distribution: for each clause type, where in the document does it occur? A label concentrated after character 400 is systematically invisible.
- Coverage: what fraction of all positive labels fall beyond the truncation point? If it is 5%, the aggregate metric is only mildly misleading; if it is 35%, the model is solving a different task.
- Per-label and per-position recall, not one micro-F1. Micro-F1 is dominated by frequent early clauses and conceals the late-clause failure.
- Business cost per missed clause type, so engineering effort is allocated by risk.
- Length-stratified performance on the 8% of documents above 8,000 characters.
The judgement is that 0.31 versus 0.88 is not yet a clean comparison: it confounds position with label frequency and clause type. Stratify by label before concluding that position is the sole cause.
2. Compare three options-head+tail truncation, overlapping sliding windows with logit aggregation, and a long-context model-on quality, latency, and engineering risk.
Read the worked answer
Head+tail truncation (e.g., 256 + 256 tokens). Cost stays at one 35 ms forward pass. It recovers clauses at the document end, which is common in contracts, but still misses the middle. It is a cheap diagnostic and a reasonable interim baseline, not a complete fix.
Overlapping sliding windows. Each window costs about 35 ms, so a 3,000-character document at 1.3 tokens/char is roughly 3,900 tokens, or about 8 windows at stride 448. Naively that is 280 ms, over budget. Engineering mitigations include batching windows on the card, increasing stride, or aggregating at sentence level. Aggregation choice matters: max-pooling logits favors any positive window and raises recall but also false positives; mean-pooling dilutes localized clauses. This option has moderate quality and moderate risk.
Long-context model. Best theoretical coverage, but attention cost grows with sequence length, pretrained Chinese long-context weights may not match the legal domain, and fine-tuning memory rises sharply. It is the highest-risk option against a 200 ms budget.
Recommendation: measure head+tail first, then sliding windows with batching and a tuned aggregation. Switch to long-context only if measured recall still misses the business threshold.
3. Design the evaluation and thresholding plan so the pilot result is not repeated in production.
Read the worked answer
The pilot's core weakness is that it evaluated a truncated model on documents it could not fully see, using a random split and probably a default threshold.
Evaluation plan:
- Split by contract, not by chunk. Chunks from the same contract must not cross train and test, or leakage will inflate results.
- Stratify by document length and report metrics per length bucket, including the 8% above 8,000 characters.
- Report per-label recall at fixed precision, plus per-position recall for the first, middle, and last thirds of each document.
- Use bootstrap confidence intervals on 300-contract pilots; a single recall estimate without an interval invites overreaction.
Thresholding plan:
- Binary cross-entropy with rare labels produces poorly calibrated probabilities, so 0.5 is arbitrary. Choose per-label thresholds on a grouped calibration set using precision-recall curves and the business cost of misses versus review effort.
- Freeze thresholds before the production pilot; do not tune them on the evaluation set.
- Monitor label prevalence and score distributions after launch, because a new contract template is a distribution shift.
The judgement is that a better model with a bad threshold and a leaky split will still fail. Fix measurement first, then architecture.