Grounding on a Policy Corpus With Six Live Versions

The instruction fix will not work and will make the problem harder to debug.

A prompt sentence cannot outvote retrieved content. The model receives a superseded chapter that states $2,450 in the same authoritative register as the current one; the instruction adds a rule with no data to satisfy it. The likely outcomes are either the stale figure (retrieval wins) or a hedged non-answer - and, worse, the instruction makes the agent sound confident while still ungrounded. The root problem is that the index does not know what 'current' means.

Architecture I would build:

1. Version and effective-date metadata on every chunk, sourced at ingestion, not inferred by the model: document_id, version, effective_from, effective_to, supersedes, jurisdiction, program.
2. Retrieval filter on currency by default - retrieve only chunks where effective_to is null or ≥ today. Superseded versions stay in the index for audit and for answering 'what changed,' but never in the default retrieval scope.
3. One authoritative source per fact where possible. Income thresholds are a structured fact, not a prose fact. Put them in a Data 360 model object or a custom object and retrieve them by lookup, so the number has exactly one home. Prose is for explanation of the rule, not the value of the rule.
4. Citations in the response (document title + effective date) so a caseworker can verify in one click; this also makes staleness detectable in production.
5. Refresh cadence matched to publication cadence - a monthly index refresh against a weekly publishing cycle guarantees a window of wrongness. Change to event-driven or nightly ingestion for the policy corpus specifically.

Sequence: fix metadata and filtering first, then move thresholds to structured retrieval, and only then re-measure. Do not rewrite instructions until the corpus is honest about its own dates.

The risk: retrieval does not automatically inherit record-level access. If the agent's grounding path queries Case records under a system or integration context, a customer (or a staff user with narrow visibility) can be told the status of a case they have no right to see - a confidentiality breach that is hard to reproduce and easy to miss, because the answer looks correct.

Controls, in order of strength:

1. Do not ground the agent on the Case object at all for status questions. Status is a single-record lookup keyed by a verified identifier. Route it through a deterministic flow that runs in the user's context and returns only fields that user may see. Grounding is for knowledge, not for transactional record access.
2. If retrieval over Case data is genuinely needed (e.g., staff-facing summarization across a caseload), enforce the query-time identity: the retrieval must execute with the running user's access, and the retrieval layer must apply the same sharing rules as the UI. Test it with a user who has deliberately restricted access - not with an admin.
3. Field-level minimization. Even an authorized record contains fields the answer does not need. Return status, stage, and last-updated only; never expose case notes, SSN, or household member detail in a retrieval payload.
4. Log every retrieval with record IDs and the acting user, so an access anomaly is investigable.

Judgment to state explicitly: the presence of a Case object in the data layer is not an authorization to ground on it. Default to 'no' for transactional objects and require a documented reason plus a sharing test to change that.

Four measures, mapped to the failure they detect:

Measure	Definition	Detects	Current target
Grounded-answer rate	% of factual claims traceable to a retrieved chunk or structured record	The original defect	≥ 98% on sampled transcripts
Citation recency	% of answers citing a document with effective_to null	Stale-version leakage	100%
Retrieval precision@k	Of the top-k chunks retrieved for a labeled question set, share that are both current and on-point	Bad chunking, missing filters	≥ 0.85 on a 100-question golden set
Callback/contact rate on a topic, 7-day	Repeat contacts citing a wrong figure	Real-world impact, not proxy	Back to baseline within 30 days

Method: build the 100-question golden set from actual call transcripts, with the correct answer and the correct source document and effective date recorded by a policy analyst. Score manually for the first month, then automate citation checks. Re-run on every corpus refresh and every model change.

What I would refuse to rely on: user satisfaction on threshold questions. Customers cannot tell a $2,450 answer from a $2,780 answer until the determination arrives - by which point the harm is done. This is a domain where automated citation checking is more trustworthy than human sentiment.