The Champion-Challenger Calibration Drift
A mid-sized bank's probability-of-default model for SME lending has been in production for 14 months. Development validation reported AUC 0.81 and a predicted one-year default rate of 1.2%. In the latest quarter, the observed default rate is 1.9%, while AUC on labeled loans has slipped only to 0.80. Population stability index is 0.28 for debt-to-income ratio, 0.21 for sector code, and 0.04 for loan size. The model owner proposes an immediate full retrain on the last quarter's data. The risk committee asks whether a calibration-only update would suffice and what monitoring should change. The bank files quarterly model performance reports, and policy requires escalation when PSI exceeds 0.25. Only 90 days of new labeled defaults are available, and the SME portfolio contains 12,000 active loans.
1. What is the most likely explanation for the gap between predicted and observed default rates, and should the committee approve a full retrain now? Justify your answer.
Read the worked answer
Diagnosis. The stable AUC (0.81 to 0.80) suggests discrimination or rank-ordering is largely intact; the problem is calibration. A predicted 1.2% versus observed 1.9% default rate is an underprediction of roughly 58% relative (0.7/1.2), which matters for pricing, capital, and loss reserves. The PSI pattern supports a covariate shift: DTI PSI 0.28 is above the 0.25 escalation threshold, and sector PSI 0.21 is near it, while loan size is stable. This points to portfolio mix or economic stress rather than a broken model.
Retrain decision. A full retrain on only 90 days of labeled defaults is risky. With 12,000 loans and a 1.9% default rate, there are only about 228 defaults, and a short window may overfit to a transient macro state and discard longer-run relationships. The committee should not approve an immediate full retrain. It should first recalibrate and investigate the DTI and sector shift, then decide whether a retrain is needed after more data and root-cause analysis.
2. Recommend a calibration and validation plan, including metrics and acceptance criteria.
Read the worked answer
Use an out-of-time calibration sample, not the training sample. Fit a logistic recalibration (intercept and slope) or Platt scaling on the most recent 90 days of labeled loans, and validate on a later holdout if available.
Report:
- calibration slope and intercept;
- Brier score;
- observed-to-expected (O/E) ratio overall and by segment (sector, DTI band, region);
- AUC and KS for discrimination;
- confusion matrices at decision thresholds.
Acceptance criteria could be O/E between 0.9 and 1.1, slope between 0.9 and 1.1, no material segment with O/E above 1.25 unless justified, and stable AUC within 0.02 of the champion. Backtest on prior quarters and run a stress scenario with higher unemployment. If the recalibrated model fails segment-level criteria, do not deploy; escalate to a full redevelopment with a longer performance window.
3. What monitoring and escalation changes would you make?
Read the worked answer
Change monitoring from quarterly-only to monthly calibration tracking: O/E, calibration slope, Brier score, and default-rate forecast error. Keep PSI but add segment-level PSI and a lower early-warning threshold (for example, 0.10 amber and 0.25 red) for DTI and sector.
Also monitor data quality, approval mix, and macroeconomic variables, and add outcome monitoring with a performance lag plus a challenger model. Escalation should trigger when:
- O/E exceeds 1.15 for two consecutive months;
- PSI exceeds threshold on a material segment;
- discrimination drops by more than 0.03.
Document the recalibration, root-cause analysis, and committee decision. If drift persists after recalibration, commission a full redevelopment with sufficient defaults and a longer performance window.