Insurance Chatbot Incident: Detection, Ownership and Evidence
Aster Insurance’s “ClaimGuide” chatbot tells customers whether water damage is likely covered. On 3 March, a customer posts a screenshot showing ClaimGuide saying a burst-pipe claim is excluded; the actual policy covers it. The post is shared 4,000 times. Aster’s model inventory lists 43 systems; internal audit finds only 31 are production, 12 are unregistered “shadow AI” tools bought by business units, and ClaimGuide’s owner left in January. The vendor contract has no audit right and no incident-notification clause. Logs are retained for 30 days. The incident is not in the risk register. The board asks for a report in 10 days. The regulator’s conduct team has asked two questions about customer harm. The company has 2.1 million policyholders and a 24-hour complaint-handling target.
1. Set out the first 72 hours of incident response, including roles and the decisions that must be made.
Read the worked answer
First 72 hours: appoint an incident commander and separate roles for containment, investigation, customer remediation, legal/regulatory assessment and communications. Contain: disable or constrain ClaimGuide’s coverage answers, replace with a static policy-lookup and a human hand-off, and preserve the current model version and prompts. Preserve evidence immediately: chat transcripts, model version, configuration, vendor correspondence, and backups, because 30-day retention may already have deleted older logs. Investigate: reproduce the burst-pipe answer, identify the root cause (retrieval error, policy wording, model hallucination, or configuration), and scope affected customers by matching conversations to policy terms. Assess harm and notification: legal and compliance decide whether the regulator’s conduct team requires a formal response, whether affected customers must be contacted, and whether the vendor must be engaged. Remediate: correct wrong coverage statements, prioritise vulnerable customers, and log every customer contact. Communicate internally with a single approved statement; do not speculate publicly. Report to the board within 10 days with a timeline, harm assessment, root cause, control failures and decisions needed. The incident commander should have authority to stop the chatbot without waiting for the business owner, because the owner has left.
2. Fix the accountability and model-inventory gaps. Propose a governance structure and the minimum controls.
Read the worked answer
The accountability gap is structural, not just a missing name. Aster needs an accountable executive for AI, a named model owner for each production system, and a product owner who understands the use case; compliance provides independent challenge, and internal audit provides assurance. A RACI should distinguish: business owner (outcomes and use), model owner (performance and documentation), compliance (regulatory mapping and monitoring), procurement (contract terms), IT/security (access and logging), and the AI governance committee (risk acceptance and prioritisation). The inventory should be mandatory and tied to procurement and expense approval: no tool goes live without an entry, a risk tier, an owner, a data-flow record, and a review date. The 12 shadow tools should be triaged within 30 days: stop, restrict, or formalise. The vendor contract should be amended at renewal or earlier if possible: audit rights, incident notification within 24 hours, cooperation with regulators, data ownership, model-change notice, and termination assistance. Internal audit should test the inventory quarterly and report to the board. The goal is not a perfect list; it is that no AI system can be bought, deployed or changed without an accountable owner and a risk record.
3. What evidence should be preserved, and what should the board report say about residual risk?
Read the worked answer
Preserve: the customer’s screenshot and original message, full chat transcript, timestamps, model and prompt version, retrieval sources, policy wording in force, configuration changes, vendor communications, incident channel logs, and the decision log. Also preserve backups and hash or otherwise authenticate the evidence. The board report should state what happened, who was harmed and how many customers are in scope, what the root cause was, which controls failed (inventory, ownership, vendor oversight, log retention, incident detection), what has been contained, and what residual risk remains. Residual risk should be rated separately for customer harm, regulatory action, financial remediation, and reputational damage, with the accepted level and the accountable executive named. The report should include a timeline, a corrective-action plan with owners and dates, and the KRIs that will show whether the fixes work: percentage of AI systems inventoried, incident detection time, vendor contracts with notification clauses, log retention compliance, and complaint rate. It should also state what the board is being asked to decide, for example, funding for a formal AI inventory and a vendor-contract remediation programme. Avoid claiming the incident is closed until the root cause is verified and affected customers are remediated.