Drift Alerts, Delayed Labels, and Retraining Policy
A lender runs a credit approval model on 300,000 applications per month. Requests and responses are logged to an inference table. A dashboard alerts when population stability index (PSI) for any feature exceeds 0.10. Income PSI has exceeded the threshold every week for two months because contractor income is seasonal; the team wants to retrain daily. True default labels arrive about 45 days after application. Current AUC is 0.78 overall, but the team has not checked segments. Employment length is missing for 15% of applicants, up from 4% last year. The model is retrained monthly on a fixed calendar, and the last two retrains were deployed without a shadow period. The team needs an alerting and retraining policy that catches real degradation without churn from seasonal drift.
1. Design the monitoring metrics and alert logic. Which signals should drive investigation versus automatic retraining?
Read the worked answer
PSI is a diagnostic, not a retraining trigger by itself. Monitor prediction-score distribution, top-feature drift, missingness, and performance once labels mature. Use the inference table for request and response profiling, and add slice metrics by product, region, channel, and income type. Income seasonality should be compared with the same month in prior years or a seasonal baseline, not a flat 0.10 threshold. Missing employment length rising from 4% to 15% is a data-quality incident; alert on it separately and check the upstream ingestion job. Alert for investigation when a feature PSI is sustained and accompanied by score shift, missingness, or segment performance drop. Automatic retraining should not fire on one weekly PSI breach. Lakehouse Monitoring can profile the inference table, and custom metrics can track approval rate and matured-cohort AUC.
2. How should the team handle 45-day label delay when evaluating the model and deciding whether recent drift is harmful?
Read the worked answer
Labels are immature for recent applications, so those cohorts cannot be scored as if they were complete. Join labels when they arrive, maintain a label table keyed by application ID and label date, and evaluate only matured cohorts. Use cohort-based AUC, calibration, approval rate, and default rate, and track the label delay itself. Do not treat unlabeled applications as negatives or impute default labels. For early warning, use validated proxy outcomes such as manual-review results or early delinquency, but confirm them against eventual labels. Compare current immature cohorts with the same stage of prior cohorts, not with fully matured ones. The 45-day delay also means a daily retraining loop would be reacting to unlabeled traffic; monthly retraining remains reasonable until matured performance shows degradation.
3. Define a retraining and promotion policy that uses drift and delayed performance without daily churn. Include canary or shadow requirements.
Read the worked answer
Keep monthly scheduled retraining as the baseline. Add an off-cycle trigger only when matured-cohort AUC drops beyond a pre-agreed threshold, for example more than 0.03 absolute for two consecutive cohorts, and a drift or missingness signal confirms the input change. Do not retrain daily on income PSI alone. When retraining, train a challenger, run it in shadow or a small canary, and compare performance on matured labels, score distribution, approval distribution, and slice metrics. Promotion requires non-inferiority on key segments and an improvement or no degradation overall. Fix the employment-length data quality issue before treating the resulting model change as a true improvement. Keep the prior Champion version available for rollback and document the trigger, evidence, and approver for each promotion.