Drift, Fairness, and a Regulator Letter
Halcyon Insurance runs a fraud-scoring model over 1.8 million claims a year. Its monitoring dashboard tracks uptime, latency, and error rate. It does not track input distributions, decisions by segment, or outcomes. Over six weeks the decline rate for claims originating in one metropolitan region rose from 8% to 19%, and complaints from that region tripled. The model has not been retrained for 14 months. Confirmed-fraud labels arrive about 90 days after the claim, so nobody can yet say whether the model is more accurate or merely harsher. Privacy blocked storing raw claim text; only aggregated numeric features are retained. The state insurance regulator has sent a letter asking for evidence of ongoing monitoring and model risk controls under its AI bulletin. The incident-response plan has no AI playbook; the on-call rotation is infrastructure engineers.
1. Is this an incident? Set out first-week actions and make the rollback decision, explaining how you decide with limited evidence.
Read the worked answer
Yes. A tripled complaint rate and a segment decline rate moving from 8% to 19% is potential consumer harm and possible unfair discrimination, which is a model-risk event even before accuracy is known.
First week: declare an AI incident and assign an accountable owner, freeze retraining, and check the data pipeline before the model. A vendor change to address normalisation, a remapped region code, or a new feed can move a segment's scores without the model changing at all, and that is the cheapest root cause to find. Run segment-level analysis of inputs, scores, and decisions; manually review a sample of 100 declined claims from the region; and check whether any threshold or override changed.
Rollback: if a previous pinned model version exists, revert it and measure. If not, apply the least invasive correction - a temporary segment-level threshold adjustment or mandatory human review for that region - rather than continuing unchanged. Do not wait for 90-day labels. Appeal rates, override rates, and manual-review disagreement are available now and are legitimate proxies. Record the decision, the evidence, and the review date, because the regulator will ask how the choice was made.
2. Design a monitoring programme that works with 90-day label latency and a prohibition on retaining raw claim text.
Read the worked answer
Use three layers. Leading indicators: population stability and KS tests on each numeric feature by segment, data-quality checks, score distribution, decision and override rates, complaint and appeal rates, and manual-review disagreement. These detect change within days without needing labels.
Delayed outcomes: build a label-maturation pipeline that evaluates cohorts at 30, 60, and 90 days against a frozen baseline, so accuracy and calibration gaps are measured when labels land rather than never. Supplement with nearer-term proxies such as SIU referral outcomes and early fraud confirmations, and validate that the proxies track the eventual labels.
Fairness: store protected attributes separately under strict access control and compute only aggregate approval-rate and error-rate gaps, with a minimum cell size so no individual is re-identifiable. That respects the privacy constraint without abandoning fairness testing.
Add shadow scoring for a challenger model, alert thresholds with named owners, and a quarterly review that feeds the model inventory and tiering. Monitoring that only sees uptime and latency is availability monitoring, not model risk management.
3. What should Halcyon put in its response to the regulator, and what assurance changes should follow?
Read the worked answer
The response should be evidence-led: the model's purpose and tier, the governance body that owns it, the monitoring actually in place and when it was last reviewed, the timeline of the drift and when it was detected, root cause if established, the containment and rollback decision with rationale, affected population estimates, and a dated remediation plan with owners. Concede the gap - the dashboard monitored availability, not model behaviour - and show the corrected design. Regulators respond better to a credible remediation plan than to a claim that controls existed when they did not.
Assurance changes: complete the model inventory and tiering so every model has an assigned owner and validation cadence; institute independent validation at least annually and on material change; formalise change control for data feeds and thresholds; add fair-lending and segment testing to the standard validation pack; and write an AI incident playbook with severity levels, a decision authority for rollback, and on-call training so the first responder is not an infrastructure engineer improvising. Report progress against the plan on a fixed schedule.