Chatbot Misstatement and AI Incident Classification
At 09:10 on a Tuesday, a bank's customer-facing chatbot began telling customers that overdraft fees would be refunded automatically if they replied 'REFUND'. The bank has 2.4 million retail customers. The chatbot had been updated two days earlier with a new prompt template. By 11:30, 3,200 customers had received the incorrect statement; 14 posted screenshots on social media. No PII was exposed, but the bank's contact center is overwhelmed and the regulator's incident portal requires notification of 'material operational incidents' within 72 hours. The AI product owner wants to issue a correction in the chatbot and continue running it. The CISO wants to shut it down. The AI governance lead must classify the incident and recommend immediate actions and governance follow-up.
1. Classify the incident and recommend immediate actions.
Read the worked answer
Classification: at least a high-severity operational and conduct incident, not a data breach. Harm is financial misstatement to 3,200 customers, contact-center overload, and public screenshots. There is no PII exposure, but the bank has made a commitment it may not honor, creating potential restitution and regulatory risk. The 72-hour notification clock should be assessed now, not after root cause.
Immediate actions:
- Invoke the AI incident response plan and appoint an incident commander.
- Use the kill switch or restrict the chatbot to static FAQ responses until the prompt template is corrected and tested.
- Preserve logs, prompts, model version, and change tickets.
- Prepare a customer correction and contact-center script, with legal and compliance approval.
- Assess whether the 14 social-media posts and customer reliance trigger regulatory notification.
- Do not let the product owner issue a chatbot-only correction without human review.
The CISO's shutdown instinct is safer than continuing, because a live system can keep generating new commitments.
2. What governance failures likely contributed, and what follow-up should the AI governance lead require?
Read the worked answer
Likely failures: no pre-deployment test of the new prompt template for policy commitments; no human approval gate for customer-facing changes; no model inventory or change record linking the prompt version to the chatbot; no defined severity matrix for AI conduct incidents; and unclear ownership among product, compliance, and incident response.
Follow-up:
- Add prompt and model changes to change management, with test cases and sign-off.
- Require a 'customer commitment' test suite before release.
- Define AI incident severity and notification criteria.
- Add a kill switch owned by operations.
- Train product teams on the escalation path.
- Report the incident to the AI governance committee and board risk committee.
- Track remediation actions to closure and test whether the correction reached all 3,200 customers, not just those who saw the chatbot.
The key lesson is that a chatbot prompt change is a production change with customer-facing consequences, not a content edit.