Default Model That Looked Excellent Until Production

The gap is too large to explain by ordinary drift. The most likely causes are leakage and sampling error:

• A random split can put the same borrower, loan vintage, or economic period in both training and validation. With 18 months of originations, random splitting also mixes the promotional underwriting period into validation instead of testing it out of time.
• days_past_due, collection_agency_flag, and last_payment_status are post-origination servicing fields. They are updated after default behavior begins, so they leak the label. At application time they are unavailable or have different meaning, which collapses top-decile lift in production.
• ROC-AUC is insensitive to the 3.1% imbalance and to ranking quality in the high-risk tail. A model can look strong on AUC while being poorly calibrated for a 0.5 cutoff.

A validation design that would expose this: split by origination month, not randomly (for example train on months 1-12, validate 13-15, test 16-18); purge any observation window that overlaps the label horizon; rebuild every feature as of the application timestamp; and add a feature-availability audit that records source system, refresh cadence, and whether the value would exist at scoring time. Report PR-AUC, top-decile lift, calibration, and expected-loss curves in addition to ROC-AUC. The production pattern, especially the collapse in top-decile lift, is consistent with leakage rather than a subtle population shift.

Class imbalance should be handled primarily through the decision rule and calibration, not by naive resampling. If resampling is used, recalibrate on the original 3.1% prevalence before setting a threshold.

Build the cost matrix: false negative = $4,200, false positive = $180, a ratio of about 23:1. For a candidate threshold t, estimate expected cost per 1,000 applications as:

FN_rate(t) * 4200 * 31 + FP_rate(t) * 180 * 969

because about 31 of 1,000 applicants default and 969 do not. Choose the threshold that minimizes this out-of-time cost, subject to compliance and capacity constraints. A 0.5 cutoff is a business convention, not an optimum. With a 23:1 ratio, the cost-minimizing threshold will usually be well below 0.5, and the model should be evaluated on approval rate and expected loss rather than accuracy.

For reason codes, use application-time features only. Post-origination servicing fields cannot appear in an adverse-action explanation. Use local explanations from a transparent challenger or model-agnostic method, map them to approved adverse-action categories, and validate that explanations are stable and plausible to compliance. Monitor approval rates and reason-code distributions by segment; a sudden concentration can signal proxy discrimination even when protected attributes are absent.

Before launch, require a model card and feature lineage register: each feature's source system, owner, refresh schedule, point-in-time availability, transformation logic, and known quality issues. Record the training window, vintage exclusions, label definition, split logic, calibration method, threshold rationale, and cost assumptions. Preserve a reproducible training pipeline and an out-of-time test set that is never used for tuning.

Escalate the post-origination feature leakage first. It is not a monitoring issue; it means the reported performance is not evidence that the model works at application time. Until the model is rebuilt with point-in-time features and revalidated out of time, launching at a 0.5 cutoff would expose the lender to uncontrolled default losses and to reason codes that cannot be defended. The missing lineage and departed builder raise reproducibility risk, but leakage is the immediate blocker.