Choosing between a hosted model API and a trained model for document review
A legal-tech team wants to extract clauses and risk flags from 200,000 contracts per month. Documents average 30 pages and are confidential. Today two analysts read samples manually. The team is deciding between calling a hosted large language model API per document and training a custom model on their labeled clause data (about 5,000 labeled examples). They also want a chat assistant for lawyers to ask questions across a matter's documents. Budget is tight and the deadline is six weeks.
1. Recommend an architecture. Justify the split between hosted model usage and custom training, and address confidentiality.
Read the worked answer
The two use cases have different economics and risk profiles, so treat them separately.
Bulk clause extraction (200k docs/month). This is high-volume, repetitive, and has labeled data. A hosted general model called per document is the fastest path to production and needs no training, but at this volume per-call cost and latency dominate, and sending 30-page confidential contracts to a third-party endpoint raises a data-handling question that must be resolved before any pilot. The pragmatic plan: start with a hosted model through a managed platform to establish a baseline and measure accuracy against the 5,000 labeled examples, while confirming the provider's data-use terms and, if required, using a deployment option that keeps data within the team's controlled environment. If volume cost or accuracy proves unacceptable, fine-tune or train a smaller model on the labeled set, which also reduces per-document cost at steady state.
Lawyer chat assistant (per-matter Q&A). This is retrieval-heavy, not training-heavy. Build retrieval over the matter's documents and have a hosted model answer grounded in retrieved passages, with citations. Do not train a model for this in six weeks.
Given the deadline, the defensible sequence is hosted model plus retrieval first, measure, then invest in custom training only where the numbers justify it.
2. What would make you reverse the recommendation and lead with a custom-trained model?
Read the worked answer
Lead with custom training if the evidence points that way:
- Volume economics: if measured per-document hosted cost at 200k/month exceeds the amortized cost of training and serving a smaller model within a few months, training wins on cost.
- Accuracy ceiling: if the hosted model's clause extraction on the labeled set misses the required accuracy and the errors are systematic (domain-specific clause phrasing), fine-tuning on the 5,000 examples is the direct fix.
- Data residency: if confidentiality rules forbid sending contract text to an external endpoint at all, a model trained and served inside the team's controlled environment becomes mandatory rather than optional.
- Latency/throughput: if batch processing windows cannot tolerate hosted API rate limits, self-hosted inference may be required.
The reverse is also true: if the labeled data is noisy, the clause set changes often, or the six-week deadline cannot absorb a training and evaluation cycle, hosted model plus retrieval remains the lower-risk choice. The decision should be driven by a measured pilot on the labeled set, not by preference for either approach.