Credit Model Drift and a Challenger Model at Larkspur Credit Union
Larkspur Credit Union uses a gradient-boosted PD model for unsecured personal loans. At validation, AUC was 0.78 and approval rate 42%. Six months later, AUC is 0.77 but approval rate has fallen to 31%, and 60+ days delinquency in the new-vintage thin-file segment is 4.9% versus 3.1% expected. Population stability index on 'monthly income' is 0.31 (above 0.25 threshold), and on 'employment tenure' 0.12. The model uses 18 features, including bureau score, income, and device type. A challenger model using cash-flow data from open banking has AUC 0.81 but only 4 months of data. The lender has a fair-lending obligation and must give adverse action reasons. Marketing wants to relax the cutoff to hit volume targets. Risk committee meets in one week.
1. Diagnose the likely causes of the approval-rate decline and delinquency rise. Distinguish data drift, concept drift, and macro effects, and state what analyses you would run.
Read the worked answer
Do not treat stable AUC as evidence that the model is fine. AUC measures ranking; approval rate and delinquency also depend on the cutoff, the applicant mix, and calibration. The income PSI of 0.31 shows a material shift in the applicant population, while employment tenure is stable at 0.12. That points first to covariate shift, not necessarily to a broken relationship between features and default.
Analyses. (1) Compare score distributions and approval rates by month, including override rates and cutoff changes; a 42% to 31% drop may be policy-driven, not model-driven. (2) Run calibration tests: predicted versus observed default by score band and by segment. (3) Build vintage delinquency curves for thin-file and non-thin-file applicants; compare 4.9% to 3.1% after controlling for score and income. (4) Test segment-level AUC/KS and feature importance stability. (5) Check whether the macro environment changed (unemployment, income volatility) and whether the model was trained through a full cycle. (6) Perform reject inference or at least assess whether the approved population is now truncated in a way that distorts monitoring.
Reasoning. If calibration is poor but ranking is stable, recalibration may be enough. If thin-file performance has deteriorated after controlling for score, the feature relationships may have changed, which is concept drift and requires retraining or a segment overlay.
2. Recommend whether to recalibrate, retrain, or deploy the challenger model. Include governance and fair-lending checks.
Read the worked answer
Recommendation: do not deploy the challenger as the sole production model. It has only 4 months of data, no full credit cycle, and open-banking coverage may be biased toward applicants who share bank data. A higher AUC in a short backtest is not sufficient evidence of stability or fairness.
Near-term path. Recalibrate the current model if ranking is stable but predicted PDs are miscalibrated, and add a thin-file segment overlay or tighter cutoff until performance is restored. Retrain with recent data only if validation shows the feature relationships have changed and the sample is adequate. Run the challenger in shadow mode, with monthly performance and fairness monitoring, before any production decision.
Governance checks. Independent validation, documented model change approval, backtesting through stress scenarios, sensitivity to cutoff changes, and monitoring for disparate impact by prohibited basis where data are available. Adverse action reasons must remain explainable; a complex challenger using cash-flow data may be harder to explain than the current 18-feature model. Marketing's volume target is not a risk rationale: relaxing the cutoff without repricing or capital adjustment would increase expected loss and may create fair-lending exposure if it affects protected groups unevenly.
Reasoning. The decision should be driven by calibration, stability, explainability, and fairness, not by a single AUC comparison.
3. How should management respond to marketing's request to relax the cutoff to hit volume targets?
Read the worked answer
Treat it as a risk-appetite decision, not a marketing decision. First, quantify the effect: if the cutoff is relaxed, estimate the increase in approvals, expected PD, expected loss, capital consumption, and delinquency by segment. Stress the result under a higher-unemployment scenario. Second, check whether the relaxation would disproportionately affect thin-file or protected groups and whether adverse action reasons remain valid.
Conditions for approval. Relaxation could be approved only with compensating controls: risk-based pricing, lower loan amounts for marginal segments, a hold-out monitoring plan, and a board-approved temporary limit. It should not be approved simply because the model's AUC is stable.
Reasoning. Volume targets create incentives to dilute credit standards. The risk committee should require evidence that the incremental volume is within risk appetite and that the fair-lending and explainability consequences are understood. If the evidence is not available in one week, defer the decision rather than approve on optimism.