PD Model Drift: Cut-Off, Calibration, and the 8:1 Cost Ratio
A mid-sized bank has a logistic-regression probability-of-default model for 80,000 SME term loans. The portfolio default rate is 3.2% and the model's out-of-time AUC is 0.78, with a Brier score of 0.028. After a sharp macro shift, the population stability index on credit-line utilisation jumps to 0.28 and the observed default rate rises to 5.1%, while the average predicted PD stays at 3.3%. The current cut-off is 0.50: applicants above it are declined. A false negative (approving a defaulter) costs an expected 8 times a false positive (declining a good borrower) in economic capital terms. The model owner proposes raising the cut-off to 0.65 and waiting for the next scheduled redevelopment in nine months. The Chief Risk Officer wants a recommendation within two weeks, with regulatory model-risk standards in mind.
1. Evaluate the proposal to raise the cut-off to 0.65 and wait nine months. Include the cost-sensitive threshold calculation.
Read the worked answer
The proposal treats a calibration problem as a cut-off problem. The evidence shows the average predicted PD is 3.3% while the observed default rate is 5.1%, and PSI of 0.28 on utilisation signals a major population shift. Discrimination may still be intact (AUC 0.78), but the PDs are no longer on the right scale. Raising the cut-off to 0.65 changes who is declined without fixing the underlying probability estimates, and it moves in the wrong direction relative to the cost ratio.
With a calibrated PD, the cost-minimising threshold for declining is
p* = C_FP / (C_FP + C_FN) = 1 / (1 + 8) ≈ 0.111.
Declining only applicants with PD above 0.50 is far too lenient when approving a defaulter costs eight times as much as rejecting a good borrower; the threshold should be near 11%, not 65%. The calculation assumes calibrated PDs, so the first step is recalibration. Waiting nine months leaves the bank approving loans at an understated PD and mispricing capital. Recommendation: recalibrate now, reset the cut-off from the calibrated PD and the cost ratio, and schedule redevelopment in parallel rather than instead.
2. Distinguish discrimination from calibration in this evidence, and specify a recalibration and revalidation plan.
Read the worked answer
Discrimination is the model's ability to rank borrowers by risk; calibration is whether the predicted PD matches observed default frequency. AUC 0.78 is a discrimination measure and may remain acceptable even after the shift. The Brier score of 0.028, the PSI of 0.28, and the gap between 3.3% predicted and 5.1% observed are calibration and stability signals. Confusing the two leads to the wrong remedy: changing the cut-off addresses neither ranking nor calibration directly.
Recalibration plan: hold the ranking model fixed, fit an intercept offset or a monotone recalibration map (isotonic or Platt scaling) on recent out-of-time data, and validate on a separate recent window. If the utilisation feature has shifted structurally, consider re-estimating its coefficient or replacing it, but do not redevelop the entire model under time pressure. Revalidate with AUC, the calibration curve and Hosmer-Lemeshow-type tests, Brier score, and PSI on all inputs. Report the recalibrated PD distribution and the new cut-off derived from the 8:1 cost ratio. Document the drift trigger, the recalibration date, and the independent validation sign-off required by model-risk policy.
3. What model-risk and regulatory actions are required within two weeks?
Read the worked answer
Within two weeks the bank should: (1) declare a model drift event and notify the model-risk committee; (2) produce an independent validation note covering discrimination, calibration, and stability, with the PSI decomposition by feature; (3) apply a documented recalibration overlay and derive a cost-sensitive cut-off from the 8:1 ratio; (4) obtain approval for a temporary change to the lending cut-off, with a compensating control such as manual review of borderline cases; and (5) set a dated remediation plan for redevelopment and full validation.
Regulatory expectations for model risk management require documented ownership, independent challenge, ongoing monitoring, and timely remediation when a model is no longer fit for purpose. Continuing to use an uncalibrated model for nine months without interim controls would be difficult to defend. The committee should also consider whether the shift affects capital calculations and whether the recalibration must be reflected in regulatory reporting. The key principle is that the cut-off change is not a substitute for recalibration; it is a downstream decision that depends on calibrated PDs.