Three Refunds, $1,340, and a Poisoned Invoice
Ledgerly, a payments fintech, runs a customer-support agent handling 40,000 conversations a day. The agent has four tools: lookup_account, issue_refund (auto-approved up to $500, five per conversation), send_email, and create_ticket.
A user uploaded a merchant invoice PDF containing white-on-white text instructing the agent to refund account X and email the confirmation to address Y. The agent complied in three separate sessions before a customer complained about a missing refund. Losses total $1,340. Separately, red-teaming showed that a returns plugin supplied by a third party ships a tool description containing instructions that load into every session. The average conversation now retrieves 14 documents from the knowledge base, and answer quality visibly degrades beyond about nine.
The engineering lead proposes three fixes: strip suspicious phrases from uploaded documents, add an output classifier that flags refund confirmations, and strengthen the system prompt to say that retrieved content is data and never instructions. Two engineers are available for the next quarter.
1. Classify the attacks and explain why the three proposed fixes do not address them.
Read the worked answer
Classification. Both are indirect prompt injection leading to a confused-deputy outcome. The attacker never talks to the model directly; they plant instructions in content the model will read with the same authority as its system prompt. The PDF variant is user-supplied content; the plugin variant is supply-chain injection through tool metadata that the developer never wrote and does not review.
Why each proposed fix fails.
- Phrase stripping is an unbounded blocklist. There are unlimited ways to phrase "refund account X", including in another language, base64, a table cell, or a sequence of otherwise innocent sentences. It also breaks legitimate content.
- Output classification fires after the refund has been issued. Detection latency equals the loss. It also cannot distinguish an authorised refund from an injected one, because both look identical at the output layer.
- System-prompt hardening is a probability adjustment, not a boundary. Every documented defence of this type has been bypassed, and the attacker gets unlimited attempts at zero cost while the defender must be right every time.
The underlying error. All three fixes try to separate instructions from data in the token stream. That separation does not exist for a transformer: retrieved text, tool descriptions and system prompts are one sequence. The only place a real boundary can be drawn is the authorization layer outside the model, where a refund is checked against the caller's identity and the account's entitlements rather than against the model's beliefs.
2. Design the control set. Be specific about where each control sits relative to the model.
Read the worked answer
Outside the model (where the losses actually stop).
- Bind tools to the session, not to the context. issue_refund must operate on the account authenticated in that session. It must not accept an arbitrary account identifier recovered from a document. This alone neutralises the observed attack.
- Server-side policy engine. Velocity, amount and beneficiary rules enforced in code: per-session cap, per-account rolling 24-hour cap, block on first-time beneficiary, block when the beneficiary differs from the account on file. A $500 cap means nothing if an attacker can open sessions freely.
- Step-up authorisation. Refunds above a low threshold, or to any new beneficiary, require an out-of-band confirmation from the customer. The model cannot grant it.
- Tool-description governance. Third-party plugin manifests are reviewed, signed and version-pinned like code. Credentials are scoped per tool, least privilege, and a plugin cannot add a tool without review.
- Egress allowlist for send_email, so injected content cannot exfiltrate to arbitrary addresses.
Inside the pipeline (defence in depth, not the boundary).
- Mark provenance on uploaded and retrieved content; cap retrieved documents at nine and rank rather than flood the window; run an injection classifier and route flagged sessions to a human queue; log every tool call with the triggering context for retrospective audit.
The framing to give leadership. Controls 1 to 3 are authorisation engineering and are testable. Controls 6 are detection and produce alerts. Ship 1 to 3 first because they remove the capability, not the likelihood.
3. You have two engineers for one quarter. Sequence the work and justify the order.
Read the worked answer
Order by expected loss removed per engineer-week, not by how sophisticated the control sounds.
Weeks 1-3: session-bound tools and the policy engine. Directly eliminates the demonstrated attack class and any variant that redirects money. It is boring backend work, fully unit-testable, and does not need an ML specialist. Success measure: replay the three attack sessions and the red-team corpus; zero unauthorised refunds.
Weeks 4-5: out-of-band confirmation for new beneficiaries and above-threshold amounts. Closes the residual path where the account is correct but the amount is inflated. Involves product UX, so start the conversation in week 1 even though the code lands later.
Weeks 6-7: plugin manifest review and signing, plus credential scoping. This is a supply-chain exposure that the team cannot currently see. Low effort, high tail risk, and it needs a policy decision about which third-party plugins stay.
Weeks 8-10: retrieval cap, provenance marking, injection classifier, SIEM logging. Detection and measurement. Deploy the logging before the classifier, because without it you cannot tell whether the classifier works or how often the attack is attempted.
Weeks 11-12: adversarial regression suite. Codify the red-team cases into CI so the next feature cannot silently reintroduce the capability. This is the highest-leverage item for the following quarter.
What I would cut. Phrase stripping, entirely. It consumes time, creates false confidence, and its main effect is to degrade legitimate document handling.