Retraining a credit model when the newest labels have not matured
A mid-sized lender retrains its application-default model on the first of each month. Default is 90+ days past due within 12 months of origination, so the earliest delinquent signal appears around day 30 and a label is not final until roughly 13 months after application. The training table carries a label-maturity column, and the most recent 13 months of applications are partially censored. The June run used applications through the end of May, but only about 4% of the last 60 days had matured at training time. Separately, population stability on the income-band feature jumped to 0.28 in May against a 0.05 baseline, and the team's first instinct is to retrain immediately including June data. A product manager suggests using 'at least one 30-days-past-due event' as a fast proxy label so the newest cohort can be included. About 2.1% of applications carry outcome 'pending'.
1. Is immediate retraining the correct response to the PSI of 0.28? Explain what PSI does and does not tell you, and what you would check first.
Read the worked answer
PSI measures input shift, not performance loss. A PSI of 0.28 on income band says the applicant population on that feature has moved. It says nothing about whether the model's ranking ability has degraded, and nothing about whether the shift is real. Two of the most common causes of a PSI spike are a data-quality artifact and a harmless shift in a feature the model barely uses.
Check, in this order:
- Is the shift real or a recoding artifact? Income bands get rebucketed by upstream teams more often than populations move. Compare raw income percentiles, not band counts. If the buckets changed, there is no drift at all.
- Feature importance. If income band carries 1% of model gain, a 0.28 PSI is nearly irrelevant. If it carries 20%, it matters. PSI is not weighted by importance and treating all features equally is why drift alerts generate so much noise.
- Realized performance on the freshest fully matured cohort. Labels through roughly day-45 are usable. Compute AUC and calibration on the most recent matured three months and compare against the training-time estimate. If AUC is within about 0.02 and calibration slope is near 1, there is no evidence the model needs replacing.
- Score distribution and approval rate versus the locked reference.
Only if step 3 shows degradation is retraining justified - and even then, retraining on data whose population has shifted and whose labels are 90% immature is more likely to hurt than help.
2. Evaluate the proposed 30-days-past-due proxy label. Under what conditions would you use it, and how would you validate it?
Read the worked answer
The proxy changes the target, and that is not a free trade. The model currently predicts eventual 90+ DPD default. 'Any 30+ DPD event' is a strictly earlier and noisier event: it captures people who are late once and recover, and it misses slow-burn defaults that first go delinquent at day 80. Training on it systematically over-weights transient delinquency. The correlation between 30-DPD-any and eventual default is typically moderate, not high, so a model optimized for the proxy can degrade on the real objective with no warning in proxy metrics.
Segmental bias is the bigger risk. Payment timing differs by income band, by payroll cycle, and by region. A 30-DPD proxy label is easier to trigger for populations with irregular cash flow, so training on it can bake in a fairness problem that the original label did not have - and it would land hardest on exactly the income-band-shifted population that triggered all this.
When I would use it: as an auxiliary signal, not a replacement. Options that keep the true target primary - multi-task learning with the proxy as an auxiliary head, sample weighting where mature true labels dominate, or using the proxy only to select which recent applications to include rather than to define their outcome.
Validation: never promote on proxy metrics. Hold out the most recent fully matured three months and evaluate the candidate on true 90+ DPD. Compare AUC, calibration slope/intercept, and approval rate at a fixed threshold against the incumbent, and do the same disaggregated by income band. Require the candidate to beat the incumbent on the true label before it can shadow.
3. Design the retraining and validation protocol that accounts for label maturation. Address the pending outcomes explicitly.
Read the worked answer
Split by application date, with a maturation buffer, never at random. Random splits leak future information into the training set whenever the label window overlaps.
Protocol:
- Rolling-origin (walk-forward) evaluation. Simulate the actual deployment date. For a model meant to go live on July 1, train on applications through, say, December 1 of the prior year and test on January-March, using only records whose labels had matured by July 1. Repeat across at least six origins and report the mean and spread of AUC, not a single number.
- Maturity filter with an explicit threshold. Include a record in training only if its label is final, or if its censorship is handled. Records at day 200 with no delinquency are not negatives - they are censored. Dropping them biases toward the bad class; calling them good biases the base rate downward. Either restrict training to fully matured records (accepting staleness) or use survival analysis / a censoring-aware likelihood.
- The 2.1% 'pending'. These are not non-defaults and must be excluded from both training and evaluation, then reconciled later. Check whether pending status correlates with income band - if it does, exclusion silently shifts the population and your June cohort is not what you think it is.
- Recency weighting. With a 45-day effective label lag, the freshest usable data is six weeks old. Apply exponential decay on application date to let the model lean recent, and test the decay half-life as a hyperparameter on the walk-forward metric.
- Promotion gate. Shadow scoring for 30 days against the incumbent, with the decision based on matured outcomes, not early delinquency.