Indirect Prompt Injection in a Retail Support Agent
Nordwind Retail runs a customer-support assistant on a commercial LLM. It ingests uploaded PDFs through OCR, retrieves policy passages from a vector store, and can call order_status, refund, and address_change APIs. Refunds above €150 need a supervisor code, but the assistant obtains that code from an internal approval bot that trusts its stated reason. In a red-team exercise, 11 of 1,200 hidden-instruction PDFs (0.9%) caused unintended tool calls; one produced a €420 refund to an attacker-controlled IBAN. The LLM vendor provides no instruction hierarchy. Traffic is 40,000 chats a day, 6% with uploads. The CISO wants launch in six weeks; the DPO objects to logging full IBANs in prompts; the engineering team proposes a stronger system prompt and a blocklist of suspicious phrases.
1. Identify the primary attack path and explain why prompt-level defences alone will not close it.
Read the worked answer
The primary path is indirect prompt injection: an attacker controls text inside an uploaded PDF, that text enters the model context alongside trusted system instructions, and the model treats it as actionable. The approval bot is a second failure point: it is a confused deputy that accepts the assistant's stated reason instead of verifying the human and the transaction. Prompt-level defences reduce probability but do not create a security boundary, because the model cannot reliably separate instructions from data and blocklists are bypassed by encoding, translation, metadata, or paraphrase. The architectural fix is to treat all PDF and retrieval content as untrusted data, keep it out of the instruction channel, and enforce high-impact actions outside the model. Refunds should be capped server-side, beneficiary changes should use allowlists and out-of-band verification, and supervisor codes should be issued to a human through a separate channel. Tool tokens should be scoped to the session and action, and the approval bot should validate the transaction against policy rather than trust natural language. Logging should tokenise IBANs so forensic value is retained without exposing full account numbers.
2. Propose a control set covering prevention, detection, and response, with measurable go-live acceptance criteria.
Read the worked answer
Prevention: sanitise PDFs by stripping metadata and flattening to images where feasible; parse content into a schema and reject unexpected fields; use a policy engine outside the LLM for refund limits and IBAN allowlists; require human approval for any beneficiary change and for refunds above a low threshold; give the model read-only tools by default and issue short-lived, narrowly scoped tokens. Detection: place canary strings in system prompts and alert if they appear in outputs or tool arguments; monitor tool-call sequences for anomalous combinations; score query diversity and injection likelihood; alert on refunds to new beneficiaries or unusual geography. Response: provide a per-session kill switch, automatic rollback of pending refunds, preserved logs in a forensic store, and a tested customer-notification playbook. Acceptance criteria should be harm-weighted, not just attack-rate-weighted: for example, zero high-impact actions without human approval across 5,000 red-team attempts in at least three languages and encoded payloads; no more than 0.5% of attempts causing any tool call; detection within two minutes; and p95 latency increase under 300 ms. The 0.9% figure must be re-measured against the new architecture, not assumed to carry over.
3. A board member asks whether the 0.9% injection success rate is acceptable. How do you frame residual risk and what evidence would you demand?
Read the worked answer
I would refuse the framing that 0.9% is a property of the system. It is an observed rate under a specific test set, and the relevant question is expected harm per day. At 40,000 chats and 6% uploads, about 2,400 uploads occur daily; 0.9% would imply roughly 21 successful injections a day if the test set were representative, but success must be weighted by whether the injected action is harmful. A failed injection that produces a wrong answer is different from a €420 refund to an attacker. The board should see a harm-weighted metric: attempted high-impact actions per 10,000 uploads, blocked by policy engine, blocked by human approval, and executed. I would demand evidence that high-impact actions cannot be executed by the model alone, that the approval bot has been tested against social engineering, that detection time and rollback time are measured, and that the residual financial exposure per day is bounded and insured. I would also require a phased launch with a lower refund cap, enhanced monitoring for 30 days, and a pre-agreed pause threshold. Without those, the rate is not acceptable because the control architecture, not the percentage, determines the loss.