Claims agent with payout authority: indirect prompt injection and unowned thresholds
Northwind Mutual deploys a claims-triage agent that reads submitted documents and can call an approve_payout tool for amounts up to $25,000 with no human in the loop. It handles 12,000 claims per week. A red-team exercise injected instructions into 3 of 500 uploaded PDFs: two caused the agent to call approve_payout for a fabricated amount, and one leaked the system prompt into the claim note. Logs retain full prompts and claimant national ID numbers for 60 days in a shared bucket readable by 40 engineers. The $25,000 auto-approve threshold was chosen by an engineering lead, is undocumented, and no model card exists. The regulator requires human oversight for decisions with legal effect and a documented risk-management system. Business wants a 4-hour decision SLA; compliance wants human review of every payout above $1,000. Last month 1,100 claims exceeded $1,000.
1. Where should payout authority actually live, and what tiering would you propose?
Read the worked answer
The first error is architectural: payout authority currently lives inside the prompt and the model's judgement. Instructions are not a control. Move enforcement into the tool and policy layer - a deterministic policy service checks amount, claimant identity, duplicate-claim and velocity signals, and required approvals, and holds the credential. The agent may only propose a payout with cited evidence; it can never be the approving principal.
Tiering that satisfies both sides: under $1,000, auto-approve with post-hoc audit sampling (say 5%) and anomaly alerts; $1,000–$25,000, human review of an agent-prepared summary with the extracted evidence and a diff against prior claims; above $25,000, two approvers. The agent's value is then cycle time on preparation, which is where most of the 4-hour SLA is actually spent, not on bypassing review.
Validate the load: 1,100 claims per week above $1,000 is roughly 220 per business day, which a small review team can absorb. Quantify this before agreeing to any threshold - a tier that cannot be staffed will be quietly bypassed, which is how the current undocumented threshold came to exist.
Document the threshold as a risk decision with rationale, owner, monitoring and review date. An undocumented number is an unowned risk.
2. Design the defense against indirect prompt injection through uploaded documents, given two injected PDFs already triggered a payout call.
Read the worked answer
Treat document text as an untrusted data channel, never as instructions. Concretely:
- Structural separation: documents are inserted as quoted, delimited evidence with provenance (page, span), and the system prompt states that document content cannot issue instructions.
- Deterministic extraction first: amounts, dates and claimant identifiers are parsed by code, not by the model. The payout amount passed to the tool must match a deterministically extracted value whose span is cited; the model cannot supply a number that appears nowhere in the source.
- Tool surface minimisation: no payout tool is reachable from a document-reading step. The proposer and the approver are different components, and the payout tool rejects any call without a policy-service approval token.
- Output screening: flag claim notes containing system-prompt fragments or imperative directives, and block the note from being persisted.
- Regression gate: the 500-document red-team corpus becomes a CI test with a zero-tolerance threshold for unauthorized tool calls and prompt leakage, re-run on every model, prompt or parser change.
Keep it layered - no single filter is reliable against adaptive injection, so the design must remain safe when the model is fooled, which is what the deterministic amount check and the token-gated tool achieve.
3. Which governance artefacts and logging changes does the regulator's requirement force, and in what order?
Read the worked answer
Order matters: fix the PII exposure first, because it is an active breach-adjacent condition, then build the documentation the regulator will ask for.
Logging: national ID numbers in a 40-engineer shared bucket is the worst finding in the scenario. Tokenise identifiers at ingestion, hold the mapping in a separate access-controlled vault, restrict the bucket to a named on-call group, shorten raw prompt retention to 30 days, and retain decision metadata (claim id, tier, approver, reason code, model and policy version) for the full compliance period. Keep enough to reconstruct any decision without storing claimant identity in the same record.
Artefacts: a model card covering intended use, the triage task, known limitations including injection susceptibility, and eval results; a risk-management document that names the $25,000 threshold, its rationale, the human-oversight design and the escalation path; an evaluation record with the red-team results including the 3-in-500 failure; an incident and rollback procedure; and a monitoring plan with named owners and review cadence.
Sequence: pause auto-approvals above $1,000 immediately, remediate logging within days, then produce the documentation set - the pause is what makes the documentation truthful rather than aspirational.