The 48,000-Token Policy Assistant That Got Worse as Recall Improved
A wealth-advisory firm runs an internal Policy Assistant on Databricks. Each request retrieves the top 40 chunks from a Delta Sync Vector Search index (roughly 48,000 tokens) and passes them, plus a 6,000-token system prompt, to a long-context chat model. Volume is 22,000 questions per day; p95 latency is 14 seconds and cost is about USD 0.09 per request. The corpus holds 3,200 policy PDFs with roughly 1,100 amendments, so several retrieved chunks often describe the same policy at different versions. Metadata fields effective_date and superseded exist, but 17% of chunks have nulls in at least one of them. Offline retrieval recall@40 is 0.94, yet answer accuracy on a 400-question human-scored set is 61%, and reviewers report answers that blend pre-amendment and post-amendment rules. An executive has proposed moving to a 1M-token model and letting the model sort out the conflicts instead of investing further in retrieval work.
1. Explain what is most likely driving 61% answer accuracy despite 0.94 recall@40, and what you would inspect first.
Read the worked answer
Recall@40 only measures presence, not decision quality. Every known-answer chunk can be present while 30+ distractors - including superseded versions of the same policy - are also present. Inspect in this order:
- Classify 40-60 failing answers by cause: missing evidence (retrieval failure), conflicting evidence (version failure), or misread evidence (packing/instruction failure). The distribution determines the fix; without it you are guessing.
- Rank position of the correct chunk in failing cases. If the right chunk sits at rank 25-38, retrieval is fine and reranking plus a smaller k is the fix. If it sits at rank 40+, chunking or metadata is the problem.
- Null rate among retrieved chunks, not corpus-wide. A 17% null rate can become 40% in retrieved sets, because older documents often match differently. Null superseded means the filter cannot exclude them, so both versions reach the model and it averages them.
- Prompt ordering: in a 48k-token pack, evidence in the middle is measurably under-used. Check whether citations show the model citing early or late chunks disproportionately.
Expect the dominant cause to be conflicting evidence from null metadata, not weak retrieval. The recall number is telling you the corpus is reachable; the accuracy number is telling you the candidate set is polluted and oversized.
2. Design the change to retrieval and context packing, including how to handle missing effective_date and superseded values, and how you would prove improvement before rollout.
Read the worked answer
1. Fix version metadata before retrieval. Backfill effective_date and superseded from document headers, sidecar JSON, or filename conventions. Where it still cannot be resolved, tag version_status = unknown rather than leaving null - you need a value the filter can act on.
2. Filter, then collapse, then rank. Hard-filter on effective_date <= question_date and superseded = false. Collapse to one chunk set per policy_id, keeping the latest effective version. Route unknown-status documents down a separate path with an explicit prompt instruction to flag uncertainty, rather than mixing them into the main candidate pool.
3. Shrink and rerank. Retrieve 20 candidates, rerank to 6-8. Pack each chunk with a header (policy_id, version, effective_date, page) so the model can attribute and detect conflicts. Place the highest-ranked evidence first and last; instruct the model to answer only from the provided blocks and to name the version it used.
4. Prove it with ablations on the frozen 400-question set, measuring answer accuracy, version-conflict rate, citation precision, tokens per request, and p95. Run k=8/20/40 with and without the metadata filter and with and without reranking. Require conflict rate to drop and accuracy to rise before shadowing live traffic.
5. Keep the switch in config, not code, so rollback is a config change rather than a redeploy.
3. The executive argues a 1M-token model removes the need for this work. How do you respond, and under what conditions would you run the pilot?
Read the worked answer
Argue from the failure mode, not from cost alone.
The contradiction does not disappear with a bigger window - it scales. The current pack already contains both versions of most amended policies; the model produces blended answers because nothing tells it which version governs. Feeding 20x more context adds more version conflicts, more mid-context evidence that gets under-weighted, and no authority signal. Bigger windows do not add metadata.
Cost and latency move the wrong way. Prefill dominates latency for retrieval-augmented chat; 48k tokens already costs ~USD 0.09 and 14 seconds p95 at 22k requests/day. A substantially larger pack multiplies both. Quantify it: tokens per request x price per token x daily volume, and show the annualized delta against the engineering cost of the metadata fix.
What the long-context model is genuinely good at is synthesis across a small, curated, version-consistent set - which is exactly what the fix delivers.
Conditions for a bounded pilot: complete the metadata backfill first, then compare at 8k vs 32k tokens on the same frozen set, with the same filter and reranker, measuring accuracy, conflict rate, p95, and cost per answered question. If the larger pack wins on accuracy at acceptable cost, adopt it. This frames the disagreement as an empirical question with a cheap, two-week test rather than a strategy debate.