Multi-Agent Assistant Exposing Internal Tools and Sensitive Data

The core flaw is that authorization is delegated to the model and a single shared role. The model is not a security boundary; any instruction-following agent can be steered by prompt injection. Identity and authorization must be enforced by deterministic code outside the model.

Flaws:

1. One IAM role for all tools and all users means any tool call has broad permissions. No per-user authorization.
2. The model decides which customer ID to pass; nothing binds the call to the authenticated user's entitlements.
3. PII in prompts flows into logs and into conversation storage without application-layer protection.
4. No input/output guardrails against prompt injection or data exfiltration.

Target architecture:

• Per-user identity propagation. Authenticate users via the corporate IdP. Pass a signed user context (subject, entitlements) to a tool-orchestration layer. The orchestration layer, not the model, injects the authorized customer scope into each tool call. The model may request 'look up this customer,' but the layer rejects requests outside the user's entitlements before the tool runs.
• Per-tool, per-user least privilege. Each tool has its own execution role scoped to specific actions and resources. Use session policies or scoped credentials so a tool call only sees the data the requesting user is entitled to. Never let the model supply raw IAM credentials or arbitrary resource identifiers.
• Guardrails. Apply input guardrails for prompt-injection and jailbreak patterns, and output guardrails for PII and sensitive-data filters. Deny-by-default tool allowlists and schema validation on tool arguments.
• Data protection. Encrypt DynamoDB with KMS customer-managed keys; encrypt sensitive fields at the application layer. Redact PII before logging; use structured logs with tokenized identifiers. Set log retention and restrict log access.
• Audit. Record every tool invocation with user, tool, arguments (redacted), and decision outcome in an immutable audit trail.

Validate with red-team cases that replay the original attack and assert denial, plus tests that PII never appears in logs.

Assume injection will sometimes succeed at the model layer, so defense must be layered and detectable.

Prevention layers:

1. Deterministic authorization. Tool calls are mediated by an orchestration service that checks the authenticated user's entitlements against the requested resource. The model's output is a proposal, not an authorization. This alone neutralizes the demonstrated attack.
2. Tool argument validation. Enforce strict schemas; reject free-form resource identifiers the model invented. Bind customer IDs to the session's authorized scope.
3. Input guardrails. Screen user input and retrieved content for injection patterns and instruction-override attempts. Treat retrieved documents as untrusted data, clearly delimited, and instruct the model to never follow instructions found inside tool results or documents.
4. Least-privilege tools. Read-only tools by default; write actions (email send) require explicit user confirmation and are rate-limited.
5. Output filtering. Block responses that contain data outside the user's scope or match PII patterns not expected for that user.

Detection:

• Log tool-call proposals and outcomes with user, tool, and a risk score. Alert on patterns: repeated denied calls, a user requesting many customer IDs, tool calls immediately after suspicious input, or calls to tools unrelated to the conversation topic.
• Track guardrail intervention rates and denial reasons; sudden spikes indicate probing.
• Correlate with identity signals (new device, unusual time) for step-up authentication.

Avoiding false positives: Tune thresholds against a labeled set of legitimate and attack sessions. Route borderline cases to human review rather than hard-blocking. Provide a clear user-facing message and an appeal path for denied actions. Run periodic red-team replays as regression tests so new prompt patterns are caught before release.