Recall halves after an embedding model upgrade in the policy assistant

A 0.84 → 0.52 drop after an embedding swap with unchanged chunks points to the embedding pipeline, not to the corpus. Check in this order:

1. Query/document asymmetry. Many embedding models expect an instruction prefix on queries (for example query: or a task description) and none on documents. If the old pipeline applied it and the new one does not, queries land in a different region of the space and recall collapses exactly like this.
2. Distance metric and normalization. Confirm the collection uses the metric the model was trained for (cosine vs inner product vs L2) and that vectors are L2-normalized if required. A rebuilt collection with the wrong metric degrades silently.
3. Dimension and truncation. If the model's max input is shorter than 512 tokens plus prefix, passages are being truncated, so the embedded text is not the text being cited.
4. Top-k and reranking. Recall@10 measures the candidate set. Without a reranker, a noisier candidate set reaches the generator.

Remediate in cost order: fix prefixes and metric, then add hybrid BM25 + dense retrieval with score fusion, then add a cross-encoder reranker over the top 50. Only then re-chunk. Re-chunking 1.2M passages is expensive and should not be the first move; it also masks a configuration bug you will hit again at the next model upgrade.

Treat tables and prose differently. Use a layout-aware parser that emits each table row as its own passage with the header row prepended and the table title and section retained, so a retrieved row is self-describing. Keep prose chunks at 400–600 tokens with 10–15% overlap, split on headings rather than fixed windows where the structure exists.

Metadata is where the versioning problem is actually solved. Attach to every passage: document ID, policy title, effective date, superseded date, status (current/superseded/draft), jurisdiction, section path, source URL, and page. Retrieval applies a metadata filter for status = current and effective_date <= today, so a superseded policy can only surface if the user explicitly asks for historical versions. Deduplicate near-identical passages across versions at index time.

For refusal, do not rely on the generator's judgment alone:

• Require a minimum retrieval score or reranker score before generation; below it, return a not-found response with the closest matching policy titles.
• Require span grounding: each factual sentence must map to an extracted quote from a retrieved passage; unsupported sentences are dropped.
• Validate that every cited document is in the current-status set before returning.

Log retrieval scores, filters applied, and refusal reasons so you can tune thresholds from real traffic.

Build a retrieval-specific regression gate that runs on every index, embedding, or chunking change, separate from the end-to-end answer-quality evaluation.

Golden set construction: 400 questions is workable but must be stratified by policy domain, question type (lookup, comparison, multi-hop), and difficulty. Each question is labeled with the passage IDs that are sufficient to answer, plus the correct current policy version. Include 40–60 deliberately unanswerable questions to test refusal.

Metrics to gate on:

• Recall@10 and Recall@50 against labeled passages.
• Version correctness: fraction of retrieved passages that are current-status when the question is about current policy.
• Table-answer accuracy: questions whose answer lives only in a table.
• Refusal precision and recall on the unanswerable subset.

Set thresholds from the pre-change baseline and block promotion on regression beyond a tolerance (for example, recall down more than 3 points). Record every run as an evaluation artifact attached to the model and index version in the governance factsheet, so the change history is auditable.

Finally, sample 50 production answers per week for human review, focusing on citations and refusals; offline recall will not catch a filter bug that only triggers on new effective dates.