PD model drift and a blunt overlay
An SME lender's 12-month probability-of-default model was developed on 2019-2022 data with an AUC of 0.81 and a KS of 0.46. On 2023-2024 vintages, AUC has fallen to 0.73 and the population stability index is 0.31; the 30-plus days-past-due rate among new loans is 4.8% versus a 3.2% forecast. The model team proposes a 15% uniform overlay to all PDs and a six-month delay in recalibration because of a vendor migration. The vendor supplies industry codes, but 12% of new applications now have unknown sector, up from 3%. The model drives pricing, limit setting, and automated approvals up to $250,000. The regulator's model-risk guidance requires ongoing monitoring and documented remediation. The CRO wants to keep approval automation unchanged to protect growth.
1. Evaluate the proposed 15% uniform overlay and six-month delay. What would you do instead?
Read the worked answer
A uniform 15% overlay is too blunt. It may close the average gap, since 4.8% versus 3.2% is a 150 basis point absolute underprediction and roughly a 50% relative increase, but it misprices low-risk and high-risk segments. A PSI of 0.31 is well above a typical 0.10-0.25 action threshold, and the AUC drop from 0.81 to 0.73 indicates both discrimination and calibration deterioration. The 12% unknown sector rate is a data-quality problem, not just a modeling problem; treating unknown as a valid category or using a fallback model is better than ignoring it.
The six-month delay is hard to defend when the model already underforecasts delinquencies and drives automated approvals. Recommended action: immediate segment-level overlays by sector, vintage, and unknown-sector flag; tighten approval cutoffs; and start recalibration now using 2023-2024 data with a documented vendor-data bridge. Run a challenger model with unknown sector as an explicit category and compare. Set a 60-90 day remediation plan, not six months.
2. How should automated approvals and pricing change while the model is remediated?
Read the worked answer
Suspend straight-through automation for the segments where the model is least reliable: unknown sector, new vintages, high PD, and exposures near the $250,000 limit. Route those to manual review or a conservative rules-based scorecard. Reduce maximum automated limits, shorten tenor, and add risk-based pricing rather than keeping the old price. Use the overlay for pricing and provisioning, but not as the sole approval gate. For low-risk, stable sectors with complete data, automation can continue under tighter monitoring.
Answer the CRO's growth argument with risk-adjusted return: underpricing 150 basis points of PD destroys capital and invites regulatory action. Track approval rates, delinquency by segment, and override rates weekly. Document the temporary control as a risk mitigation with an owner and expiry date.
3. What model-risk and data-governance steps are needed to satisfy the regulator and restore the model?
Read the worked answer
Treat this as a model-risk event: document the drift, root cause, impact, and remediation; obtain independent validation; and report to the model-risk committee and board. Data governance: profile the vendor feed, quantify unknown-sector by channel, add a data-quality dashboard, require fallback treatment, and set a service-level agreement with the vendor.
Recalibration: use recent vintages, out-of-time and out-of-sample tests, segment-level calibration comparing observed and expected defaults, and backtesting of approvals. Monitor PSI, AUC, KS, and delinquency gaps monthly with escalation thresholds. Validate the overlay and challenger model and define rollback conditions. Communicate proactively with the regulator if the model is material to capital or lending decisions. Restore automation only after acceptance criteria are met, such as AUC of at least 0.78, PSI below 0.15, and calibration within tolerance for two consecutive quarters.