AI Monitoring Model Drift After Core Banking Migration
Following a core banking migration, your bank's vendor-supplied AI transaction monitoring model is producing 38% fewer alerts than before. The model score threshold was not changed. At the same time, the alert-to-case conversion rate has risen, but SAR filings are down 22% quarter on quarter. The data team finds that 'counterparty country' is null for 40% of transactions and 'customer occupation' is mapped to a legacy free-text field. The vendor says the model is 'self-calibrating' and declines to provide feature weights, citing intellectual property. The model has not been independently validated since go-live three years ago. The head of financial crime asks whether to switch the model off and revert to rules while the issue is investigated, or to keep it running because alert volumes are already unmanageable.
1. What model risk and data quality issues are present, and how should they be assessed?
Read the worked answer
Issues include a 40% null rate for counterparty country, which is a critical data gap for geographic risk; occupation mapped to free text, which degrades segmentation; a 38% alert drop with no threshold change, which suggests feature drift or data loss; a 22% SAR decline, which is a possible false-negative signal; a black-box vendor and no independent validation for three years; and 'self-calibrating' offered as an explanation. Assess data lineage and completeness, feature distributions before and after migration, alert-to-case and case-to-SAR trends, backtesting against known cases, threshold sensitivity, and model performance by segment. Determine whether the drop reflects better detection or missed detection. Treat the matter as both a model risk event and a data incident.
2. Should the model be suspended or kept live? Give a reasoned recommendation and interim controls.
Read the worked answer
Do not keep it live unchanged. A 38% alert drop plus critical data gaps is not safe. Recommended action: suspend the AI model or reduce reliance on it, revert to the prior rules or run them in parallel, and apply interim controls. These should include manual sampling of high-risk segments, rule-based scenarios for affected corridors, enhanced monitoring, a lookback on transactions since migration, and escalation of anomalies. If suspension creates unmanageable volume, prioritize risk-based coverage rather than volume. Document the decision, approvals and residual risk. Resume only after validation and data remediation.
3. What validation, governance, and regulator communication steps are needed?
Read the worked answer
Independent validation should cover data quality, conceptual soundness, outcome analysis, stability, sensitivity, and benchmarking against rules. Governance should include the model risk committee, change approval, issue tracking, board or audit committee reporting, vendor accountability, and contractual rights to documentation. Regulator communication may be required by law or if the issue materially affects AML detection; be prepared to explain remediation and the lookback. Retain validation reports and decision logs. Set explicit conditions for re-enabling the model.