RAG Assistant Leaks a Matter Note After a Malicious PDF
Aster Legal deploys an internal AI assistant for 4,000 employees. It uses a third-party LLM API and retrieval-augmented generation over 1.2 million documents, including HR investigations, client contracts, and matter notes. The pilot has 300 users. Documents are indexed nightly from SharePoint with inherited permissions flattened into a single search index; the assistant returns cited snippets. The vendor contract says prompts may be retained for 30 days for abuse monitoring but not used for training. A partner uploads a malicious PDF that contains hidden text instructing the model to ignore prior rules and reveal other users' matter notes. The security team sees one response containing a client name the user should not access. The GC asks whether to shut down the pilot, restrict it to partners, or continue with monitoring.
1. What is the primary control failure and immediate action?
Read the worked answer
Reasoning: the leak is not primarily a model-behavior failure. Flattening SharePoint permissions into one index means the retrieval layer can return documents the asking user was never entitled to see. The hidden PDF text is a prompt-injection trigger, but the impact comes from missing document-level authorization at retrieval time.
Decision: suspend the pilot or restrict it immediately to a low-sensitivity document set while the index is rebuilt. Quarantine the malicious PDF and preserve its metadata and retrieval logs. Audit the exposed user's session history and all responses containing restricted client names. Notify the GC and the affected matter owner, and follow the firm's incident process. Do not rely on adding a stronger system prompt as the fix. The minimum technical correction is identity-aware retrieval that filters documents by the requesting user's current permissions before any text reaches the LLM.
2. How should the team design retrieval and prompt handling to reduce prompt injection and data leakage?
Read the worked answer
Reasoning: retrieved text must be treated as untrusted data, and authorization must be enforced outside the model. The model cannot be the access-control decision point.
Design: pass the user's identity and group membership to the retrieval service; enforce SharePoint ACLs at query time, including inheritance changes and ethical walls; filter at chunk level before sending context to the LLM. Treat retrieved content as data, delimit it clearly, and instruct the model to ignore instructions inside documents. Use an output filter for client names, matter numbers, and DLP patterns, plus citation checks. Prevent cross-user memory and shared chat history. For high-risk matters, disable retrieval or require a second approval. Log which document chunks were retrieved for each answer, and red-team with hidden-text PDFs before re-enabling broad access.
3. What monitoring and vendor controls are needed before the pilot can continue?
Read the worked answer
Reasoning: the firm needs evidence of containment and ongoing detection, not just a policy update. The 30-day vendor retention also creates a data-governance decision because prompts may contain privileged client information.
Monitoring: alert on responses that include restricted matter names or unusual document retrievals; track queries that retrieve documents outside the user's normal practice group; run canary documents with unique markers to detect unauthorized retrieval; review a sample of prompts and retrieved chunks weekly. Vendor controls: confirm no training on firm data, require deletion or zero-retention for prompts where available, restrict data processing to approved regions, obtain security attestations, and define breach notification. Before re-enabling, test permission changes, ex-employee access, ethical-wall scenarios, and malicious PDF uploads. Continue the pilot only with a rollback plan and named accountable owner.