The Redaction Model That Was Fine Yesterday
A hospital's radiology department runs a locally hosted LLM to draft imaging reports. Before deployment, the team ran a 500-prompt red-team suite and the model leaked protected health information (PHI) in 3 of 500 responses (0.6%), which the risk committee accepted as within tolerance. Six months later, a vendor-supplied retrieval-augmented-generation (RAG) component is added so clinicians can query prior reports. No new evaluation is performed. A nurse notices the assistant quoting a different patient's name and MRN in a draft. Logs show the RAG index was built from a nightly export that includes records the model was originally never trained on, and the export job was never scoped to the requesting clinician's patient panel. The CISO wants to either pull the feature or add a guardrail before Monday.
1. The team argues the original 0.6% leakage rate proves the model is safe and the new incident is a fluke. Evaluate this reasoning and identify the actual control that failed.
Read the worked answer
The reasoning is unsound because it treats a pre-deployment measurement as a permanent property of the system. The 0.6% figure characterized a model evaluated against a fixed prompt suite with no retrieval. Adding RAG changed the system's data flow: the model can now surface content it never saw during evaluation, so the original metric no longer describes the deployed system.
The failed control is authorization scoping on the retrieval corpus / data pipeline, not the model's safety tuning. The nightly export pulled records outside the requesting clinician's patient panel, so the retriever returned documents the user was never entitled to see. No amount of model-level red-teaming would have caught this, because the leak originates upstream of the model.
Correct framing: this is a broken access-control boundary in the RAG data layer. The model faithfully summarized documents it should never have received.
2. The CISO proposes a post-filter that scans outputs for MRN patterns and blocks them. Assess whether this is sufficient and state what must accompany it.
Read the worked answer
A pattern-based output filter is a useful defense-in-depth layer but is not sufficient and would create false assurance if treated as the fix.
Why insufficient:
- PHI leaks in many forms the regex won't catch: names, dates of birth, free-text identifiers, or paraphrased details that re-identify a patient without an MRN string.
- It does nothing about the root cause - the retriever still hands unauthorized records to the model. A determined or unlucky query still gets the content; only the final string is scrubbed.
- It cannot distinguish a legitimately authorized MRN (the clinician's own patient) from an unauthorized one, so it will either over-block or under-block.
What must accompany it:
- Fix the data boundary: scope the RAG index/export to the requesting clinician's authorized patient panel, enforced at query time via access-control metadata on each document, not by filtering after retrieval.
- Re-run evaluation against the new system, including retrieval-specific adversarial prompts, and set a new acceptance threshold.
- Logging and alerting on cross-panel retrieval attempts so the incident class is observable.
The filter is a seatbelt, not the brake.
3. Recommend the decision the CISO should make for the Monday deadline, and justify the risk trade-off.
Read the worked answer
Recommendation: disable the RAG feature until the authorization scoping is implemented and re-evaluated; do not ship the post-filter alone as a stopgap for PHI exposure.
Justification:
- The exposure is unauthorized disclosure of PHI across patients - a reportable, high-severity harm with regulatory and patient-safety consequences. The probability is not theoretical; it has already occurred in production.
- A post-filter does not close the boundary, so residual risk remains high and hard to bound. Shipping it as "the fix" would misrepresent the risk posture to the committee.
- The original 0.6% acceptance was for a different system; there is no valid current risk measurement to justify continued operation.
Path to re-enable: implement document-level access control in retrieval, re-run the red-team suite plus retrieval-specific cases, confirm cross-panel leakage is zero (not merely low), then restore with the output filter as an additional layer. If clinical need is urgent, a limited pilot scoped to a single clinician's own panel is the defensible middle ground.