Leakage and Censored Vintages in an SME Credit Model
A fintech lender originating working-capital loans to small businesses wants to replace its scorecard with a gradient-boosted model. The analytics team assembles 2019–2023 originations: 84,000 loans, 11% twelve-month default at 90+ days past due. Two warehouse fields rank at the top of feature importance: collections_status (current, 1–30 DPD, 31–60, and so on) and payment_plan_active. Mid-2022 the credit policy tightened - minimum revenue rose and two industries were excluded - cutting approvals by 38%. Labels mature twelve months after origination, so recent vintages are incomplete. The head of credit wants a single holdout AUC and a go-live in six weeks; the model will run at application time from an API that receives only the applicant's submitted data plus a bureau pull. Regulatory review requires documented model risk management.
1. Explain why the two highest-ranked features are problematic, how you would verify the problem, and how you would fix it.
Read the worked answer
Both fields are recorded after an adverse event. A loan that will default acquires a populated, deteriorating collections_status; a performing loan stays "current." A model trained on them effectively learns "if a collections record exists, default" - a rule that cannot be computed at application time because the API receives no servicing history.
Verification. For every candidate feature, record the system timestamp at which the value first becomes available and compare it to the credit decision timestamp. Any feature populated only after origination is leakage by inspection. A faster check is to retrain without the two fields: a large AUC drop confirms they were carrying the signal. Also look for softer leakage - a bureau score refreshed after origination, a "final industry code" assigned during underwriting review.
Fix. Build a point-in-time feature table as of the application date rather than snapshotting the current warehouse row. Drop payment_plan_active outright: it exists only for existing customers and reflects post-origination servicing. Then re-examine importance - a model whose top features vanish is a model that never had legitimate signal, and that is better discovered now than in production.
Document the finding in the model risk file: it demonstrates that controls worked.
2. Design the validation scheme given the mid-2022 policy change and immature 2023 vintages. Be specific about what you would train on, what you would test on, and what you would report.
Read the worked answer
Random k-fold is wrong twice over.
Temporal. The 2022 tightening changed the approved population - smaller, higher-revenue, two industries removed. A random split lets the model borrow information across regimes and overstates performance on applicants it will actually see.
Censoring. 2023 originations have not had twelve months to default. Coding "not yet defaulted" as "good" makes recent vintages look artificially clean.
Scheme. Build vintage cohorts by origination month. Train on 2019–2021, validate out-of-time on 2022 H1, test on 2022 H2–2023 using only matured loans, or apply survival methods that treat censored observations correctly. Report AUC, KS, and - most importantly for a lending decision - calibration, broken out by vintage and by the segments affected by the policy change (revenue band, the two excluded industries as a sanity check on extrapolation).
Approved-only sample. You only observe borrowers you approved; the score must rank the marginal applicant, whom you have never seen. Apply reject inference or, at minimum, document the truncation and the assumption explicitly. An unstated assumption is the one that gets a model pulled.
3. The head of credit insists on a six-week go-live with the leakage fixed and "a holdout AUC." What do you agree to, and what do you put in writing?
Read the worked answer
Six weeks is enough to build. It is not enough to validate a model that will become the primary credit decision.
What I agree to. Go live as a challenger score: it runs alongside the existing scorecard, is logged, and binds nothing. Pre-register the evaluation window and metrics - approval-rate impact at constant bad rate, calibration by vintage, and score stability on the live applicant flow. That serves the business urgency without staking the portfolio on a model that has not seen out-of-time data.
What I put in writing. The regulator's expectation is a documented chain: purpose, data lineage, methodology, validation results, known limitations (approved-only sample, censored vintages, the leakage finding and its remediation), and ongoing monitoring with named owners and thresholds. I would flag the timeline risk explicitly, with a list of what will not be finished by week six and the consequence of skipping it - not a vague objection, a schedule.
Why bother. Documentation is not paperwork here. The leakage discovery is exactly the kind of evidence that a model risk function exists to produce, and recording it protects both the team and the firm.