Eleven Days of Silence: A Fraud Model That Changed Behaviour
Cobalt Bank uses a vendor-supplied real-time fraud scoring model. On a Tuesday the vendor pushes version 4.3 under a "continuous improvement" clause with no notice. Two days later an upstream team changes the merchant-category feed, which begins emitting nulls; the model imputes zero and the effect is masked.
Within 72 hours, false declines on small-business debit cards rise 3.1x. The call-centre queue grows 40%. Every dashboard stays green because they monitor latency, error rate and infrastructure. The prediction-distribution shift appears in a data-quality table that nobody alerts on. A risk analyst notices on day eleven: 2,300 customers affected, 340 complaints, 2 already escalated to the regulator.
Cobalt's incident response plan is written for data breaches. There is no AI incident taxonomy, no model owner on call, and no agreed definition of what counts as a model incident. The vendor's contract requires notification of "security incidents" within 24 hours, which nobody believes covers this.
1. Is this a reportable incident? Reason through the classification rather than answering yes or no.
Read the worked answer
Start by rejecting the framing in the question. The team is asking "is this a breach?" because that is the only trigger the plan recognises. The correct first question is "what kind of harm occurred?" Here the confidentiality dimension is untouched and the integrity and availability dimensions are badly damaged: the model produced wrong decisions at scale for eleven days.
Work the regimes.
- Operational resilience / outsourcing rules (EU DORA and equivalents). Major-incident criteria include number of clients affected, duration, geographic spread, economic impact and criticality of the service. Two thousand three hundred customers, eleven days and a regulator escalation will meet several of these. If DORA applies, the clock runs from detection and the classification decision itself must be recorded.
- GDPR. This is not obviously a personal-data breach, so Articles 33 and 34 may not trigger. But Article 22 does: significant effects from automated decisions with no meaningful human review. Document the human-review design now, because that question will be asked.
- Conduct and fair-treatment rules. Wrongful declines against small businesses are a consumer-harm issue in most jurisdictions and often the fastest route to a supervisory conversation.
- Contract. "Security incident" almost certainly does not cover this, which is exactly the gap to close.
The defensible position. Classify it as an AI incident with integrity impact, record the reasoning for whichever notification decision is taken, and notify the vendor regardless of the clause's wording. An undocumented decision not to notify is the worst outcome available.
2. Design the detection layer that would have caught this in 24 hours. Specify signals, thresholds and ownership.
Read the worked answer
Tier 1: hourly technical signals, owned by the platform on-call. Latency and error rate are necessary and were never sufficient. Add: null rate and schema fingerprint per feature, with a hard alert above 2% for any feature the model weights materially; inference-time schema mismatch; and null volume on the merchant-category feed specifically, which was the actual trigger.
Tier 2: daily model behaviour signals, owned by the model owner. Prediction distribution (PSI or KS against the reference window), approval/decline rate, score-band migration, and feature-attribution shift. Critically, compute these per segment, not globally. Here the global decline rate barely moved because small-business debit rose while another segment fell; a Simpson's-paradox artefact hid the incident from any aggregate view.
Tier 3: business outcome proxies, owned by operations. Call-centre queue depth and wait time, complaint volume and topic clustering, manual override rate by reviewers, appeal and reinstatement rate. These are the signals that would have fired on day two without any ML tooling, and they are usually the cheapest to wire up.
Thresholds. Use change detection against a trailing baseline rather than fixed numbers, and set the alert on the rate of change so a slow drift is caught as well as a step. Route tier 1 and 2 alerts into the same on-call rotation as availability incidents; a model incident that is not paging is not monitored.
After-the-fact check. Replay the incident window through the proposed alerts and confirm they fire within 24 hours, with a false-positive rate the team can live with.
3. Write the first-60-minutes playbook for the next occurrence, and name the two contractual changes you would pursue.
Read the worked answer
First 60 minutes.
- Declare. Anyone on call can declare a suspected model incident; no approval needed to open one. The cost of a false alarm is an hour; the cost of the last one was eleven days.
- Stabilise first, diagnose second. Freeze the current model version and feature feed, or roll back to the last known-good version, if a rollback path exists that is faster than diagnosis. Confirm the rollback path exists before you need it; here it probably did not.
- Quantify the blast radius from logs: decisions affected, customers affected, money moved, time window. Assign one person to that number and nobody else.
- Open the human channel. Notify the call centre and complaints team with approved language, so front-line staff stop improvising.
- Classify against the incident taxonomy using a written checklist, and record the decision and the rationale for notification or non-notification, with a timestamp and a named owner.
- Notify the vendor in parallel and request the change log for the pushed version.
Two contractual changes.
- Change control: no model version reaches production without notice, a change note and a documented rollback to the prior version. "Continuous improvement" should mean continuous evaluation, not continuous deployment into a regulated decision.
- Incident breadth: redefine the notification clause to cover service degradation and model-behaviour incidents with defined time-to-notify, and require the vendor to supply the model version, feature dependencies and monitoring telemetry needed for the bank to meet its own regulatory clocks.
Also fix the internal gap the incident exposed: the plan's triggering definition, not just the vendor's.