The Summarizer That Hallucinates Citations
A legal-tech startup builds a retrieval-augmented generation (RAG) assistant that summarizes case law for paralegals. The pipeline retrieves the top 5 passages from an internal corpus, then prompts a large language model to 'summarize the relevant holdings and cite the source documents.' In a 200-query evaluation, 12% of summaries contained at least one citation to a case that was not in the retrieved passages, and 5% cited a real case for a proposition it does not support. The engineering lead argues this is acceptable because 'the model is just summarizing what it retrieved,' and proposes lowering the temperature to 0 to fix it. The compliance team wants to know whether the tool can be used for client-facing work.
1. Evaluate the engineering lead's claim that lowering temperature will fix the citation problem, and explain the actual failure modes.
Read the worked answer
Lowering temperature reduces sampling variance but does not eliminate hallucinated citations. The failure is not primarily stochastic: it is a grounding and attribution problem. Two distinct modes are present:
- Fabricated citations (12%) - the model produces a plausible-looking case name or docket number not in the retrieved context. This is the model's parametric knowledge and pattern-completion filling gaps; temperature 0 still does this deterministically.
- Misattribution (5%) - a real, retrieved case is cited for a proposition it does not actually support. This is a semantic entailment failure: the model over-generalizes or conflates passages. Temperature is irrelevant here.
The lead's framing ('just summarizing what it retrieved') is also wrong: the prompt asks the model to 'cite source documents,' which invites it to generate citations rather than extract them. The correct fix is architectural: require citations to be extracted verbatim from retrieved passage IDs, verify each claim against its cited passage with an entailment check, and reject or flag summaries where verification fails. Temperature 0 may reduce surface variability but leaves both error rates essentially intact.
2. Design an evaluation and gating process the compliance team could accept before any client-facing use.
Read the worked answer
Task definition: Separate the evaluation into (a) citation validity - every cited case exists in the retrieved set; (b) attribution correctness - the cited passage entails the claim; (c) summary faithfulness - no unsupported assertions.
Measurement: Build a labeled gold set of, say, 300 queries stratified by practice area and query type. Have two attorneys annotate independently and report inter-annotator agreement. Compute: citation-validity rate, attribution precision/recall, and unsupported-claim rate with confidence intervals. Track per-slice results (e.g., long documents, ambiguous queries).
Automated guardrails: Implement a citation verifier that checks each cited ID against the retrieval set and runs an entailment model (or human spot-check) on claim–passage pairs. Block output when verification fails.
Gating criteria: e.g., citation validity ≥ 99.5%, attribution precision ≥ 98%, unsupported-claim rate ≤ 1%, with a documented human-in-the-loop review for any flagged output. Require a pilot with paralegals, logging all outputs and corrections, before expanding scope.
Governance: Document model version, prompt, corpus snapshot, and evaluation date; define a re-evaluation trigger when the corpus or model changes. Present residual risk explicitly to compliance rather than claiming 'fixed.'