Model Risk in a Vendor Scorecard

Two things must be separated. Discrimination (ranking borrowers) is measured by AUC; calibration (predicted default rates matching actual) is separate. The internal AUC of 0.74 versus vendor 0.81 is a meaningful drop but still indicates some discriminatory power. The more serious issue is calibration: defaults at 1.8x predicted on recent cohorts means the model systematically understates risk, which directly affects pricing, provisioning, and approval thresholds. The cohort pattern matters. If performance degrades specifically on recent originations, that is consistent with either population shift (new borrower types), environmental change (macro conditions), or the vendor's sample not representing the company's portfolio. 'Seasoning' is a plausible partial explanation-recent loans have less time to default-but 1.8x is large and the validation team should test whether the gap persists as cohorts age. Fit for purpose is therefore conditional: usable for ranking if thresholds are recalibrated, not usable as-is for expected-loss estimation. The CRO should require calibration testing by cohort and a documented decision on acceptable divergence.

Opacity raises model risk because the company cannot independently verify assumptions, and vendor validation reports may be self-serving. Appropriate controls: (1) independent outcome testing-the company validates on its own data, which it already does; make this continuous and cohort-level, not one-off. (2) Calibration overlay-apply an internal adjustment factor to the vendor's outputs based on observed divergence, reviewed quarterly. (3) Challenger model-build or buy a simpler internal model to benchmark rankings and detect drift. (4) Contractual controls-require the vendor to disclose validation methodology, notify of model changes, provide change logs, and permit audit; the contract should make undisclosed material changes a breach. (5) Documentation-record the model's known limitations and the rationale for use despite opacity. (6) Concentration and exit planning-assess dependence on a single vendor and maintain a migration path. The CRO cannot eliminate opacity but can bound its consequences through outcome testing and overlays.

A defensible decision is conditional continued use, not unconditional retention and not immediate replacement. Replacement is expensive and slow, and the model still discriminates; abandoning it would create a gap. But using it as-is with 1.8x under-prediction is not acceptable. Conditions: (1) apply an internal calibration overlay to predicted default rates for recent cohorts, effective immediately, with the overlay reviewed monthly; (2) tighten approval thresholds or increase pricing for segments where divergence is largest until calibration is restored; (3) require the vendor to provide cohort-level validation data and a remediation plan within 60 days, with contractual escalation if not met; (4) commission a challenger model and set a decision date (e.g., 9 months) to compare performance; (5) report divergence and overlay impact to the risk committee quarterly. Document the decision rationale, including why replacement was deferred, so the judgement is auditable. This balances cost, continuity, and prudence, and puts a time-bound trigger on the vendor's remediation rather than open-ended reliance.