Model Validation: A Credit Scorecard With Suspiciously Stable Backtesting
A lender's model risk team is validating a new probability-of-default scorecard before deployment. The developer reports a Gini of 0.62 on the 24-month out-of-time sample and a population stability index (PSI) of 0.03 versus the development sample - both comfortably inside policy thresholds (Gini ≥ 0.40, PSI < 0.10). The validator notices the out-of-time sample was drawn from the same 18-month origination window as the development sample, just with a different random seed. Default rates in the portfolio have risen from 2.1% to 4.7% over the last year. The developer also winsorized the top and bottom 1% of the score distribution after measuring performance, and reports the post-winsorization figures. Deployment is scheduled in ten days and the business has already priced products using the model.
1. Evaluate the validity of the reported Gini and PSI as evidence of model performance.
Read the worked answer
Neither statistic supports the conclusion drawn. The 'out-of-time' sample is not out-of-time: drawing from the same 18-month origination window with a different seed produces a random holdout, which tests in-sample fit, not temporal stability. A Gini of 0.62 on a random split says nothing about how the model behaves on a 4.7% default environment when it was built on a 2.1% one. The PSI of 0.03 is likewise uninformative because the comparison population is the same window - you cannot detect population drift by comparing a sample to itself.
Winsorizing after measuring performance and then reporting the post-winsorization numbers is a reporting-integrity problem: it presents a cleaned figure as if it were the model's measured discrimination. The validator should require a genuine out-of-time sample (originations after the development window, ideally spanning the default-rate shift), recompute Gini and PSI on it, and require performance to be reported on the un-winsorized distribution with the winsorization disclosed separately.
2. Given ten days to deployment and products already priced, recommend a course of action.
Read the worked answer
The validator cannot sign off on the current evidence, but the business reality (pricing already done, tight timeline) shapes the recommendation rather than excusing it. Options: (1) block deployment pending a valid out-of-time test - safest but may strand priced products; (2) conditional/interim approval with compensating controls - cap exposure, apply a conservative overlay or margin to the PDs, monitor early performance against a defined trigger, and commit to a full revalidation on genuine out-of-time data within a fixed window.
The defensible answer weighs model risk against business disruption and documents the trade-off. A conditional approval must specify the overlay, the monitoring thresholds (e.g., observed vs. predicted default rate, PSI on live population), and the revalidation date. What is not acceptable is deploying on the strength of the random-split Gini as if it were out-of-time evidence. The validator should also escalate the post-hoc winsorization as a methodology finding, because it affects trust in the developer's other reported figures.