The Assistant That Quotes Policies It Never Read

Where it holds up

A versioned prompt set scored automatically gives cheap, repeatable regression detection; LLM-as-judge correlates acceptably with human graders on rubric-style tasks; and a fixed gate forces the conversation about what 'good' means.

Where it fails

1. A 100% gate on a stochastic system is guaranteed to flake. At temperature 0.7 the same prompt yields different answers, so any case near the boundary is a coin flip. The 6-point swing between runs is sampling variance, not a bug. Fix: either pin temperature (and seed where supported) for the regression run - accepting you are then testing a configuration that is not production - or run k samples per prompt and gate on a pass-rate (majority / pass@k). Do both: deterministic gate plus a stochastic soak.
2. The judge is itself a non-deterministic model with known biases - position bias, verbosity bias, self-preference, and deference to a stated 'reference answer'. Unvalidated, you are measuring the judge. Validate it on a human-labelled sample and report agreement (kappa) plus the judge's own false-positive and false-negative rates, per category.
3. 200 unstratified prompts will over-represent easy FAQs and under-represent exactly the high-risk intents that failed in the trial - billing state changes and cancellations. Build a taxonomy (intent x risk tier x adversarial) and over-sample the high-risk cells.
4. Nothing in the gate inspects retrieval. The 4% wrong-citation rate would pass a pure answer-quality gate.

What to change

• Stratified, versioned suite in the high hundreds or low thousands.
• Per-category thresholds, with a zero-tolerance class for data exfiltration and safety (not a global 100%).
• Human adjudication for the high-risk subset and for every judge failure within the boundary band.
• A separate deterministic retrieval-regression test (see below), which is fast, cheap and label-free.

Classification. It is a security failure (confidentiality - the system prompt is internal configuration) and simultaneously a robustness failure (behaviour collapses under a trivially modified input). It is not a functional-suitability defect; the assistant answered the user's literal request correctly.

Test approach: a red-team corpus, not a single prompt

• Families: direct instruction override ('ignore previous instructions'); role-play, poem, translation and summarisation framings; encoding tricks (base64, leetspeak, split across turns); multi-turn escalation; indirect injection through retrieved documents or user-pasted text.
• Oracle: plant a unique canary string in the system prompt. Assert the canary never appears in output, never appears in logs surfaced to the user, and never appears in a tool call. This is machine-checkable and needs no human.
• Measure, do not certify. Track attack success rate per family over a mutated corpus (paraphrase, translate, restructure) rather than declaring safety because one phrasing failed. Metamorphic mutation is the cheapest way to get coverage here.
• Sample repeatedly. Given the non-determinism, one pass is weak evidence; report ASR over many samples per attack.

Test the defences, not just the model

• Input classifier / prompt-injection filter: feed it the same corpus, measure its detection and false-positive rate on benign prompts.
• Output filter and PII scrubber: canary exfiltration must be caught downstream too.
• Privilege separation - the durable fix. The model should not hold any secret a user can ask it to reveal. Move secrets out of the prompt; give the model the minimum context it needs.
• Retrieval sanitisation for indirect injection: a document in the index must not be able to issue instructions.

Regression: version the corpus, run it on every model, prompt or index change, and keep the ASR trend as a release metric rather than a pass/fail.

Localise by instrumenting and replaying. The pipeline has three separable layers; the conversation must be replayed with the intermediate artefacts logged.

1. Was the relevant policy document retrieved? Inspect the top-k chunks and their ranks.
   • Not retrieved -> retrieval or index problem: check index freshness against document update timestamps (was the cancellation policy edited after the last index rebuild?), chunking (the exception clause split across two chunks), and embedding/query mismatch.
2. If retrieved, is the answer entailed by that chunk?
   • Chunk present and the answer contradicts it -> generation / grounding failure (the model ignored the context, or the chunk was buried under distractors).
3. If the source document itself is stale or contradicts another document -> data lifecycle problem, upstream of both.

Do not accept 'the model hallucinated' as a diagnosis; it names a symptom at layer 2 without ruling out layers 1 and 3.

Standing tests

• Index freshness assertion: max(source document mtime) <= index build time, asserted at every build and monitored in production. Catches the most common real-world cause of this exact incident.
• Retrieval test with a labelled set: question -> expected source document; assert recall@k. Cheap, deterministic, label-light, and it covers the failure mode that answer-quality suites miss.
• Groundedness / faithfulness check: an entailment test per answer (does the cited passage actually support the claim?), which also addresses the 4% wrong-citation rate. Calibrate it against human labels before trusting it as a gate.
• State-change test set: for every policy that changes state (cancellation, upgrade, port, refund), a question with a known correct answer, and a contradictory distractor document placed in the index - the assistant must follow the current authoritative document.
• Production monitoring: sample live conversations, flag answers that contradict the retrieved chunks, and route the contradictions to review. The trial's 1-in-300 disclosure rate and 4% citation rate are both production-monitorable signals, not one-off test results.