The Already-Patched Incident
A public-facing chatbot at a utilities company told 40 customers that a overdue bill could be ignored because of a 'hardship exemption' that does not exist. A engineer changed the prompt the same afternoon and the behaviour stopped. No incident ticket was raised; the change was not reviewed; the chatbot's owner says the issue is closed because the fix is live. Three of the 40 customers later received late-payment notices, and one has complained to the regulator. The AIMS internal audit is due in two weeks, and management wants to present the matter as a resolved technical bug rather than an AI incident. The chatbot's impact assessment predates the feature that allowed free-text billing questions.
1. Is treating this as a resolved technical bug acceptable? Justify your answer.
Read the worked answer
No. A same-day technical fix addresses the symptom, not the AIMS failure. The chatbot gave incorrect billing advice to 40 customers, three received late-payment notices, and a regulator complaint exists. That is an AI incident with customer harm and potential regulatory consequence.
Calling it a 'bug' obscures the management-system issues: no incident ticket, no review of the prompt change, no impact assessment for the free-text billing feature, and no evidence that affected customers were corrected or compensated. The fix being live does not show that the cause was understood or that recurrence is prevented.
ISO/IEC 42001 expects incidents and nonconformities to be logged, contained, investigated, corrected, and reviewed for effectiveness. The chatbot owner's position would leave the AIMS unable to demonstrate learning. Management should be advised that presenting it as a closed bug is likely to fail audit scrutiny and may worsen the regulatory position.
2. What corrective actions and documented information should the AIMS owner require?
Read the worked answer
The AIMS owner should require a formal incident record and corrective action, not a retrospective note.
Actions:
- Containment: identify all affected customers, correct billing records, withdraw late-payment notices, and provide a clear apology and route to redress.
- Impact assessment: assess the free-text billing feature and update the pre-existing assessment.
- Root cause: examine why the chatbot could assert a non-existent exemption, why the prompt change was unreviewed, and why no incident was raised.
- Corrective action: add change control for prompts and model configuration, guardrails for billing claims, and an AI incident category in the service desk.
- Effectiveness check: test the chatbot against billing scenarios and monitor complaints.
- Regulator handling: coordinate the response with legal and compliance.
Documented information should include the incident report, affected-customer log, root-cause analysis, change record, updated impact assessment, corrective-action plan, and effectiveness evidence. The prompt change itself should be traceable to an approved change record.
3. What should management review and the internal audit programme take from this event?
Read the worked answer
Management review should treat the event as evidence about the AIMS's effectiveness, not just a customer-service matter. Inputs should include the incident, the regulatory complaint, the missed impact assessment, the unreviewed prompt change, and the status of corrective actions. Management should decide whether resources, roles, or escalation thresholds need to change.
The internal audit programme should test whether similar conversational features elsewhere have current impact assessments and change controls. It should sample prompt and configuration changes to see whether they were reviewed and recorded. It should also check that incident reporting actually captures AI issues rather than routing them as ordinary bugs.
The broader lesson is that a fast fix can mask a slow control failure. If management review records show only 'fixed' and not 'why it happened and what changed', the AIMS will look reactive. The audit should verify both the correction and the system-level learning, and report any residual nonconformity honestly.