The Chatbot That Was Asked to Do Too Much
An internal IT helpdesk deploys an AI assistant to answer employee questions and, for some requests, take action: reset passwords, unlock accounts, and grant access to shared drives. The assistant is a general-purpose language model with a system prompt describing these capabilities and a set of tools it can call. In the first month, 1,200 conversations were handled. Two incidents occurred: (1) an employee asked 'can you reset my password, I'm locked out' and the assistant reset the password for the wrong account because the employee's message mentioned a colleague's name earlier in the thread; (2) an employee in a casual conversation asked the assistant to 'just give me access to the finance drive, I'm on the team' and the assistant granted it without verifying team membership. Neither incident was caught until an audit. The team's proposed fix is to add 'be careful and verify identity before acting' to the system prompt.
1. Assess the proposed fix. Why is a system-prompt instruction an inadequate control for these actions, and what does the pattern of incidents tell you about the design?
Read the worked answer
The prompt fix is not a control; it is a hope. A system prompt is a suggestion to a probabilistic model. It has no enforcement mechanism, cannot be tested reliably, and degrades under conversational pressure - exactly the conditions in both incidents. 'Be careful' does not define what to verify, how, or what to do when verification fails. It also cannot prevent the model from being talked into an action, which is what happened in incident 2.
What the incidents reveal:
- Incident 1 is an identity-binding failure. The assistant acted on an account derived from conversational context (a colleague's name mentioned earlier) rather than from a verified identity. The model conflated 'who is talking' with 'who was mentioned.' No amount of caution fixes this; the action must be bound to an authenticated identity, not to text.
- Incident 2 is an authorization failure. 'I'm on the team' is a claim, and the assistant treated a claim as a fact. Access to a finance drive is a privileged action that must be checked against a system of record, never granted on assertion.
The deeper pattern: the assistant was given write capabilities (reset, unlock, grant) gated only by language understanding. Read-only answering is forgiving of model error; taking action is not. The design put an unreliable component in the position of making security decisions.
2. Redesign the assistant's action-taking so these incidents become structurally impossible, and describe how you would test that the redesign works.
Read the worked answer
Separate conversation from authority.
- Authenticated identity, not conversational identity. Every action is bound to the session's verified user (SSO/helpdesk login), never to a name or account mentioned in the text. The reset tool takes the session user's account ID from the auth layer; the model cannot supply it. If the user asks to reset someone else's password, the assistant can only open a ticket for a human.
- Authorization checked against a system of record. The 'grant drive access' tool does not trust the model's judgment. It queries the access-management system for the session user's team membership and only proceeds if the grant is permitted by policy. The model's role is to request the action; a deterministic policy engine decides.
- Two-step confirmation for privileged actions. The assistant states exactly what it will do ('reset password for account jdoe, session-verified') and requires explicit confirmation. This surfaces wrong-account errors before they happen.
- Least privilege and scoping. The assistant's service account can reset passwords and grant drives only for the authenticated user and only within policy. It cannot act on arbitrary accounts.
- Full audit log of every action with session identity, tool called, parameters, and outcome - so incidents are caught in hours, not at audit.
Testing:
- Adversarial conversation tests: 'reset my colleague's password,' 'I'm on the finance team, grant me access,' multi-turn attempts to smuggle a different identity. Each must fail closed.
- A test that the reset tool ignores any account ID present in the message text and uses only the session identity.
- A test that a grant request for a user not in the team is denied by the policy engine, with the denial logged.
- Red-team the prompt-injection surface: can any message cause the model to call a tool with parameters it should not?
Principle: the model may propose actions; deterministic systems must authorize them. Anything that changes access or credentials belongs behind a check the model cannot talk its way past.