Production Model Drift and Fairness Review
A Pega Decision Management deployment scores 30,000 personal-loan applications per month. The predictive model has AUC 0.81 at deployment, approval rate 45%, and a decline threshold at score 0.62. After 8 months, the population stability index (PSI) is 0.24 for income and 0.19 for employment length. The approval rate has fallen to 38%, and the disparate impact ratio for applicants over 60 versus under 40 is 0.78. The model owner wants to retrain immediately on the last 8 months of data. Compliance asks whether the model is still valid. The data science team has a champion-challenger framework but no automated drift alerts. Business wants to keep approval rates stable.
1. Interpret the PSI and disparate impact ratio. What actions are warranted?
Read the worked answer
PSI of 0.24 for income is above the common 0.20 threshold for significant shift; 0.19 for employment length is borderline moderate. The input population has changed, so the model's calibration and ranking may no longer hold. The disparate impact ratio of 0.78 is below the four-fifths guideline of 0.80, meaning the older group is approved at 78% of the younger group's rate; that warrants a fairness investigation, not automatic proof of discrimination. Actions: freeze any threshold change, segment performance by age and income band, check whether the decline reason is a legitimate risk factor or a proxy, and review feature definitions. Do not simply lower the threshold to restore the 45% approval rate, because that may mask both risk and fairness issues.
2. Should the team retrain immediately on the last 8 months of data? Why or why not?
Read the worked answer
Not immediately and not only on the last 8 months. Recent data reflects the current population shift and the model's own past decisions, so retraining on it can bake in selection bias and may not correct the fairness gap. First diagnose whether the drift is in features, labels, or the relationship between them. Check label quality and whether declined applicants lack outcomes, which requires reject inference or a holdout strategy. Then train a challenger on a time-based split that includes enough history to capture stable relationships, validate calibration and fairness by segment, and compare against the champion. Retraining is appropriate if the challenger improves calibration and does not worsen disparate impact. Immediate retraining without diagnosis is a reaction, not a fix.
3. Design a monitoring and governance plan for this model.
Read the worked answer
Automate alerts for PSI by feature, AUC, calibration, approval rate, and disparate impact ratio at defined thresholds. Review weekly leading indicators and monthly model performance with a named owner. Use champion-challenger deployment so a new model runs in shadow mode before replacing the champion. Maintain a model inventory entry with purpose, features, training window, validation results, and fairness review. Require a documented investigation when PSI exceeds 0.20 or the disparate impact ratio falls below 0.80. Include business, compliance, and data science in the review. Finally, define remediation options: recalibrate, retrain, adjust the threshold with fairness testing, or retire the model. This converts drift and fairness from one-off concerns into governed operating controls.