The AI Chatbot That Gave Bad Advice
A UK building society launched an AI-powered chatbot to handle customer queries about mortgages and savings. The chatbot uses a large language model (LLM) fine-tuned on the society's product documentation. Within weeks, the chatbot gave incorrect advice on early repayment charges to 200 customers, some of whom incurred financial loss. The society's compliance team is concerned about breach of FCA rules on fair treatment of customers. The chatbot was developed by a third-party vendor, and the society's IT team had limited involvement. The vendor's contract includes a liability cap of £50k. The society's board is questioning whether to continue using AI. The CTO argues that the technology is sound and the issue was due to a data quality problem in the fine-tuning dataset. The CCO argues that the society should have had human oversight and testing before deployment.
1. Analyse the governance failures in the deployment of the AI chatbot and recommend controls to prevent recurrence. Consider the FCA's principles and the society's responsibilities.
Read the worked answer
Governance failures: (1) Lack of board and senior management oversight – the board was not informed of the risks. (2) Inadequate risk assessment – no assessment of customer harm. (3) Third-party risk management – the vendor was not adequately overseen; contract liability cap insufficient. (4) Lack of human oversight – no human review of chatbot outputs before deployment or in operation. (5) Data quality – fine-tuning data not validated. (6) Testing – insufficient testing of chatbot responses. (7) No model risk management framework. FCA principles: Principle 6 (Customers' interests) and Principle 7 (Communications with clients) require clear, fair, and not misleading communications. The society is responsible for the chatbot's advice even if vendor-provided. Controls: (a) Establish an AI governance committee with board representation; (b) Implement a model risk management framework including validation, testing, and monitoring; (c) Require human-in-the-loop for high-risk queries (e.g., mortgage advice); (d) Conduct thorough testing with diverse scenarios; (e) Improve vendor contracts to include audit rights, liability, and data quality requirements; (f) Implement ongoing monitoring and feedback loops; (g) Train staff on AI risks. The CTO's data quality argument is valid but insufficient; governance and oversight are essential.
2. Discuss the ethical and regulatory implications of using AI for customer-facing advice, and how the society should remediate affected customers.
Read the worked answer
Ethical implications: fairness, transparency, accountability. Customers should be told they are interacting with AI. Regulatory: FCA expects firms to be able to explain AI decisions (explainability) and to treat customers fairly. The society may have breached FCA rules and should self-report if significant. Remediation: (1) Identify all affected customers (200) and contact them proactively; (2) Reimburse any financial loss and correct any misinformation; (3) Apologise and explain what went wrong; (4) Review the chatbot's advice and correct the fine-tuning data; (5) Suspend the chatbot until controls are in place; (6) Conduct a root cause analysis and report to the board; (7) Consider whether to notify the ICO if personal data was involved. The society should also review its complaints handling process. The CCO is correct that human oversight was missing. The society should not blame the vendor solely; it retains responsibility. Going forward, the society should adopt a responsible AI policy and ensure human oversight for high-risk decisions.