Shadow AI Prompts and a Prompt-Injected Customer Assistant
Arvid Foods has 4,000 employees, and traffic logs show 1,100 of them using generative AI services, 47 of which are not on any approved list. In a three-week sample, 320 prompts contained material the data loss prevention engine classified as source code or unpublished pricing. In parallel, the data science team runs a customer-facing recipe assistant built on a hosted large language model with a retrieval layer over 60,000 internal documents, plus a demand-forecasting model trained in a managed notebook environment. A contracted researcher demonstrated that a crafted PDF uploaded to the assistant caused it to disclose its system prompt and an API key embedded in a retrieval document, and could make it call an internal pricing endpoint. Legal wants all generative AI blocked; the CISO wants the assistant to ship next quarter and the engineering teams to keep their coding tools.
1. Split this into the control planes you would present to the CISO, and state what you would do in the first 30 days in each.
Read the worked answer
There are three problems wearing one label, and each needs a different owner and budget.
Usage governance (employees using third-party AI): you cannot govern what you cannot see, so the first deliverable is a discovered inventory of AI destinations by user, volume, and data classification. Days 1-30: publish a tiered catalogue (approved tenants, approved with restrictions, blocked), enable browser-level prompt controls that detect source code and pricing patterns in submitted text, and require the enterprise tenants of approved tools where data-retention terms are acceptable. The 320 prompts are a policy and training signal first; a hard block on day one produces shadow usage over personal hotspots.
Application security (the recipe assistant you are shipping): days 1-30 are containment, not features. Rotate the exposed API key, remove secrets from the retrieved corpus entirely, restrict the assistant's outbound network path to an allow list that excludes internal pricing endpoints, and add per-user authorisation at retrieval time.
Model and data lifecycle: inventory the models, record training data provenance, classify the retrieved document store, and decide retention for prompts and completions.
What to say to Legal: blocking employee use is a policy lever; the assistant needs engineering controls, and blocking it does not remove the security work.
2. The researcher's PDF attack succeeded twice: prompt disclosure and a tool call to an internal endpoint. What architecture changes actually reduce this risk, given that prompt-injection filtering is unreliable?
Read the worked answer
Treat every model output as untrusted input to the next system, exactly as you would treat a user-supplied string.
Architectural changes, in order of effectiveness:
- No secrets the model can reach. The API key was in a retrieved document; that is a data-hygiene failure, not a model failure. Secrets belong in a vault accessed by the application under its own identity, never in a prompt, a document, or a completion.
- Authorisation at retrieval, per request, against the asking user's entitlements. Pre-filtering an index by tenant is not enough when a document can be surfaced by similarity to an attacker's phrasing.
- Least privilege for tools. The assistant called an internal pricing endpoint because it was allowed to. Give it the smallest set of read-only tools with per-tool scopes, no write actions without human confirmation, and an egress allow list at the network layer.
- Output validation before action or display: schema checks, blocked-pattern checks, and stripping of credentials, URLs, and markup.
- Human in the loop for anything irreversible: refunds, order changes, outbound email.
What none of this fixes: the model can still be persuaded to say wrong things. Filtering input and output raises cost for the attacker but is probabilistic. Design so that a successful injection yields nothing valuable. Log prompts, retrieved document IDs, tool calls, and outputs for every session so the next researcher's finding is detectable after the fact.
3. Legal wants generative AI blocked outright. How do you respond in a way that is defensible to both Legal and the engineering teams?
Read the worked answer
Agree with the underlying requirement, disagree with the instrument. Legal is asking for assurance that confidential material does not leave the company and cannot be reconstituted by a third party; a blanket block does not deliver that assurance, it relocates the activity out of sight.
The argument to make, in their language:
- With no approved path, the 1,100 users move to personal accounts and mobile hotspots. You lose the logs that produced the 320-prompt finding, which is your only current evidence base. You also lose the ability to negotiate retention and training terms with vendors.
- Regulatory exposure follows the data, not the browser. A blocked prompt and a prompt sent from a phone are the same disclosure in a breach notification.
The proposal:
- Tier one, approved enterprise tenants with no-training retention, SSO, and DLP inspection. Coding tools that pass a security review go here.
- Tier two, allowed with guardrails: redaction of identifiers, blocking of pricing and source-code patterns, no file upload of classified material.
- Tier three, blocked: consumer tools with training-on-input terms and unvetted browser extensions.
- An exception process with a named approver and a 90-day review, so a team with a real need is not forced to hide.
What you commit to in return: monthly reporting of blocked prompts, incidents, and exceptions granted, and a quarterly review with Legal. If the data shows the tiers are not holding, you revisit the policy with evidence.