A Generative Assistant That Quoted the Wrong Fees
Harbourline Bank launched an in-app generative assistant 14 weeks ago. It serves 1.6m users, handles 240,000 conversations a month, and answers using retrieval over 1,100 product pages. Compliance has now found 1,180 transcripts in which the assistant quoted an incorrect early-repayment fee, and one in which it gave a view on the tax treatment of an offset mortgage. The retrieval index includes an archived 2022 fee schedule that was never removed. Fourteen days ago the vendor pushed a model update without notice; the change is not in the model inventory and no regression test was rerun. Marketing copy on the app store describes the assistant as providing "personalised advice". The vendor contract has no change-notification clause and the service is hosted in a single region. The head of digital wants to keep the assistant live while fixes are made, citing a 22% reduction in call-centre volume.
1. Classify the incident and set out your actions in the first 72 hours.
Read the worked answer
Classification: this is one event with four exposures - operational risk (system/data), conduct risk (incorrect fee information likely to cause customer detriment), model risk (unvalidated model change and stale retrieval corpus), and outsourcing risk (vendor change without notice, single-region hosting).
First 72 hours:
- Stop the bleed. Disable generative responses on fees, tax and eligibility; replace with deterministic, approved answers and a link to the tariff. Do not take the assistant offline entirely if the fallback is safe - but the fallback must be scripted, not free-text.
- Quantify harm. Extract the 1,180 transcripts, identify customers who acted on the wrong figure and calculate actual detriment. Any customer charged the wrong fee is remediated automatically; do not wait for complaints.
- Preserve evidence. Freeze logs, prompts, retrieval index versions and the vendor's model version history before any remediation changes them.
- Escalate. Notify the chief risk officer, DPO and internal audit same day. Assess reportability to the regulator against the conduct/operational incident thresholds and the outsourcing rules; a tax view is likely unauthorised advice and may be separately reportable.
- Vendor action. Written breach notice, demand the model change record and a rollback option, and require change notice as a contract amendment. Suspend further vendor updates until governance is in place.
- Communications. Correct the app store wording immediately; it is a statement to customers and is itself a conduct issue.
2. Marketing wants to keep the "personalised advice" wording, and the vendor resists a change-notification obligation. How do you resolve both, and what control set would you require before the assistant is re-enabled for fee and tax topics?
Read the worked answer
On the wording: the phrase is not a marketing preference, it is a regulatory characterisation. "Personalised advice" implies a suitability obligation the assistant cannot discharge: it has no fact-find, no appropriateness assessment and no advice record. The wording must change to something accurate such as "general product information", with a clear statement that it is not advice and a route to a human adviser. Marketing cannot approve its own consumer-facing claims on a regulated tool; compliance sign-off is a control, not a courtesy.
On the vendor: change notification is a non-negotiable term for a model that produces customer-facing statements. If the vendor will not contract for it, the bank cannot rely on the tool for regulated content. Practical levers: make the change right a condition of renewal, require version pinning so the bank controls when updates apply, and impose a contractual regression-test obligation. If the vendor still refuses, move fee and tax content to an internally controlled retrieval layer.
Controls before re-enablement:
- Curated, version-controlled content only; retired documents removed and a test that fails on stale sources.
- Pre-deployment regression suite of at least 300 adversarial and standard prompts, rerun on every model or index change.
- Retrieval citations shown to the user and logged.
- Topic guardrails blocking tax, advice and eligibility questions.
- Human review sample of 5% of conversations weekly, with a hard stop if accuracy falls below 98% on fees.
- Model inventory entry, named owner and board-level metric on hallucination rate.
Re-enable fee topics first, with tax permanently excluded.