Drift Alert Fatigue and Delayed Labels in Fraud Monitoring
A fintech launched a card-not-present fraud model. Monitoring computes PSI on 24 features and fires about 40 alerts per day; the on-call data scientist has muted most of them. Ground-truth chargebacks arrive 45 days later, so weekly AUC is unavailable at decision time. The model was trained on 2023 data and retrained monthly. Since launch, average transaction amount rose 18%, a new merchant category grew from 2% to 11% of traffic, and the decline rate dropped from 4.1% to 3.6%. The business wants a 'drift score' on a dashboard and automatic retraining. Compliance requires that any retrain be validated and documented before promotion, and that model behavior changes be explainable. The team currently has no labeled holdout for the new merchant category and uses the same Kafka topic for training features and live scoring.
1. Diagnose why the current alerting fails and propose a monitoring metric set.
Read the worked answer
Forty PSI alerts per day on 24 features is an alerting design problem, not a drift problem. PSI is univariate and sensitive to sample size and binning; with high traffic, small irrelevant shifts cross thresholds. The on-call muted them because there is no ranking, no ownership, and no link to business impact. Replace the single PSI threshold with a tiered set: data quality and schema checks as hard alerts; feature drift ranked by contribution and persistence, not raw count; score distribution monitoring, including mean score, decline rate, and review queue volume by segment; delayed performance once labels mature; and proxy performance from manual review outcomes. Use control limits based on a stable reference window and require persistence over several days before paging. A dashboard 'drift score' is acceptable for the business only if it is backed by these components and shows which segment is driving it.
2. Design a retraining trigger and promotion workflow that satisfies compliance.
Read the worked answer
Combine triggers rather than automating on a single metric: a sustained proxy-performance drop beyond a control limit for at least five business days, a confirmed label-matured AUC drop beyond an agreed threshold, a material shift in a business-critical segment such as the new merchant category, and a maximum calendar interval. Any one trigger opens a retraining candidate, not a promotion. The candidate trains on a time-based split, is evaluated against the champion on a locked holdout, and must pass segment-level performance, fairness, and explainability checks. Run it in shadow on live traffic for a defined period and compare decisions and scores. A human owner approves promotion with a documented model card and change record; the previous version stays deployable for rollback. Automate the pipeline and the evidence collection, but keep the promotion gate human and auditable. The current monthly retrain should continue until the new trigger proves more reliable.
3. How should the team handle the 45-day label delay and the missing holdout for the new merchant category?
Read the worked answer
Treat labels as an eventually consistent signal and build three layers. First, use maturation curves to estimate what the eventual AUC will be from partial labels, and never compare a 7-day AUC to a 45-day AUC without adjusting. Second, create proxy labels from manual review decisions and early fraud confirmations; validate that the proxy tracks matured labels on historical data before trusting it. Third, keep a small, stratified human review sample for the new merchant category so the team can measure performance there rather than waiting for chargebacks. For the missing holdout, construct one retrospectively from the first weeks of the category, label it through review, and freeze it. Add segment-level monitoring so a global AUC that looks stable does not hide a collapse in the new category. If the category remains unlabeled and too small, cap exposure or apply a conservative threshold until evidence exists.