Drift in a Credit-Scoring Model After a Data Pipeline Change
A lender runs a gradient-boosted model to rank small-business credit applicants. In March, the feature team replaced a batch job that computed 'average monthly inflow' from 90-day windows with a streaming job using 30-day windows, intending to improve freshness. Nobody updated the model card or the monitoring thresholds. By June, the approval rate for applicants in the $50k-$150k revenue band fell from 41% to 29%, while the model's AUC on a held-out set stayed flat at 0.79. The risk team's dashboard shows a PSI of 0.31 on the inflow feature but no alert fired because the threshold was set to 0.35. Complaints from two broker partners mention 'sudden rejections of previously approvable clients.' The model owner argues nothing is wrong because discrimination hasn't degraded.
1. Is this a model performance problem, a data quality problem, or a governance problem? Justify your classification and identify the primary failure.
Read the worked answer
It is primarily a governance failure with a data-quality root cause, and calling it a 'performance problem' would mislead the response.
- AUC unchanged (0.79) means the model still ranks the population it was trained on well. But the feature definition changed underneath it: a 30-day inflow window is not the same construct as a 90-day window, so the model is now applying learned relationships to a differently-defined input. Discrimination on a held-out set won't catch this if the held-out set was scored with the new pipeline too.
- PSI 0.31 vs threshold 0.35 shows the monitoring was calibrated to the old feature distribution and never revisited after the pipeline change. The threshold was not wrong in isolation; it was stale.
- The real failure is that a material change to a model input was made by an engineering team without triggering the model change-management process: no model card update, no re-validation, no threshold recalibration, no stakeholder notification.
So: root cause = unmanaged upstream data change; proximate symptom = distribution shift; governing defect = change management that didn't cover feature pipelines.
2. The model owner wants to lower the PSI alert threshold to 0.25 and move on. What would you recommend instead, and why is the threshold change alone insufficient?
Read the worked answer
Lowering the threshold to 0.25 is a monitoring tuning action, not a remediation. It would have fired an alert in April, which is useful, but it does not:
- Restore construct validity. The model was trained on 90-day inflow. Either revert to a 90-day-equivalent feature, retrain on 30-day features, or formally re-validate that the 30-day construct preserves the intended risk relationship. Pick one and document it.
- Address the harmed population. Applicants in the $50k-$150k band were rejected at a materially higher rate for ~3 months. That warrants a look-back, remediation of adversely affected applicants, and a fair-lending review of whether the shift disparately impacted protected groups.
- Fix the change-management gap. Add feature-pipeline changes to the model change trigger list, require model-owner sign-off, and tie threshold recalibration to any approved feature change.
Recommended sequence: (a) freeze the streaming change or revert, (b) re-validate and decide retrain vs. revert, (c) recalibrate thresholds as part of that re-validation, (d) look-back and remediate, (e) update model card and change log. The threshold change is step (c), not the whole plan.
3. What monitoring signals, beyond PSI and AUC, should have caught this earlier?
Read the worked answer
PSI on a single feature and aggregate AUC are both too coarse. Signals that would have surfaced this in weeks, not months:
- Feature-level distribution monitoring with segment breakdowns: PSI/KS on inflow by revenue band, not pooled. The pooled PSI of 0.31 likely masked a much larger shift in the $50k-$150k segment.
- Outcome/decision monitoring: approval rate by segment, tracked as a first-class metric with control limits. A 41%→29% move is a 12-point drop and should have its own alert.
- Input schema and lineage checks: automated detection that a feature's upstream source/job changed, with a required acknowledgment before the new values flow to production.
- Prediction distribution monitoring: score distribution shift, not just AUC.
- Business/complaint signals: broker complaint text mentioning 'sudden rejections' should route into the model risk queue, not just customer service.
- Stability of feature-target relationship: monitor the learned relationship (e.g., partial dependence or coefficient/importance stability), not just rank ordering.
The principle: monitor inputs, decisions, and relationships, not only output accuracy.