Choosing Between Fine-Tuning and Retrieval for a Legal Team
A legal department wants a generative AI system that drafts first-pass summaries of 200-page contracts. Requirements: summaries must cite the exact clause text they rely on; the firm's internal style guide must be followed; the model must not reveal one client's contract details when working on another's; and the system must run in the firm's AWS account with no data leaving its control. The firm has 5,000 past contract-summary pairs and a budget for one iteration. Latency target is under 30 seconds per contract. The team is debating prompt engineering alone, retrieval-augmented generation, fine-tuning, or a combination.
1. Recommend an architecture and justify it against each requirement, including why the alternatives fall short.
Read the worked answer
A combination is the right answer, with retrieval as the backbone and fine-tuning only if needed for style.
Prompt engineering alone fails the citation requirement: the model cannot reliably quote clause text it has not been given, and with 200-page contracts the relevant clauses will not fit in context reliably. It also cannot enforce client isolation.
Fine-tuning alone is a poor fit for citation and freshness. Fine-tuning teaches patterns and style but does not guarantee that quoted text matches the source, and it does not solve per-client data isolation. It also consumes the one iteration budget on a task where retrieval delivers more.
Retrieval-augmented generation addresses citation directly: retrieve the relevant clauses, pass them as context, and require the model to cite them. Client isolation is handled by partitioning the index per client and filtering retrieval by client ID, so one client's documents are never in another's context. Running in the firm's AWS account satisfies the data-residency constraint.
Fine-tuning can be layered on the 5,000 pairs to teach the internal style guide, but only after retrieval and grounding are working, and only if prompt-based style instructions prove insufficient. Latency is achievable if retrieval is scoped to the relevant sections rather than the whole document.
2. The firm has budget for one iteration. What should they measure in that iteration to decide whether fine-tuning is needed at all?
Read the worked answer
They should spend the iteration on a retrieval-plus-prompting baseline and measure the gap to requirements, rather than starting with fine-tuning.
Measure: (1) citation accuracy - does every quoted clause match source text exactly; (2) clause recall - were the clauses that matter for the summary retrieved; (3) style compliance - scored against the internal style guide by a reviewer or rubric; (4) client-isolation tests - deliberately query with the wrong client ID and confirm no cross-client content appears; (5) latency per contract against the 30-second target.
If citation and recall are strong but style compliance is weak, that is the signal for fine-tuning on the 5,000 pairs. If style is acceptable from prompting, fine-tuning is unnecessary and the budget is better spent hardening retrieval and evaluation.
This ordering avoids spending the only iteration on fine-tuning a model that still hallucinates citations, which would leave the core requirement unmet.