The Vendor Model That Nobody Can Validate
A regional bank relies on a third-party vendor's credit-scoring model to price small-business loans. The model was purchased 18 months ago and has never been independently validated; the vendor treats the algorithm as proprietary and provides only summary statistics. In the last two quarters, the model's predicted default rate for the portfolio was 2.1% and 2.3%, while realized defaults were 3.4% and 4.1%. The model owner argues the gap reflects a weak economy, not model failure. The bank's model risk policy classifies any model used in credit decisions as high-tier, requiring annual validation and ongoing monitoring. The next validation is not scheduled for six months, and the vendor contract has no audit or validation clause. Loan volume in this segment is growing 15% per quarter.
1. Assess whether this is a model risk issue, a credit risk issue, or both, and recommend immediate actions.
Read the worked answer
It is both, and the model risk dimension is the binding one. The realized-to-predicted default ratio is roughly 1.6x and 1.8x over two consecutive quarters - a persistent, directional under-prediction, not random noise. A weak economy can explain a level shift, but a well-calibrated model should have been recalibrated or its limitations flagged; the consistent bias suggests the model is mis-specified for this portfolio or was built on a different population. Because the model drives pricing and credit decisions, the bank is systematically under-pricing risk (credit risk consequence) while operating a high-tier model without validation (model risk consequence). Immediate actions: (1) treat the model as out of tolerance and apply a conservative overlay or judgemental adjustment to pricing and provisioning until validated; (2) commission an independent validation now, not in six months, using available data and challenger models; (3) pause or cap growth in the segment until the validation concludes; (4) open a contract renegotiation to obtain validation rights. Continuing to scale a mis-calibrated model at 15% per quarter compounds the exposure.
2. The vendor refuses to disclose the algorithm. How should the bank proceed without the source code?
Read the worked answer
Non-disclosure of source code does not excuse the bank from validation; it changes the validation approach and raises the model's risk rating. The bank can validate behaviourally: back-test predicted versus realized defaults across segments and time, test discrimination (e.g., rank-ordering of borrowers), stress the model under scenarios, compare outputs to an internal challenger model, and assess data quality and the vendor's development documentation. Where the black box cannot be adequately understood, the policy should require a conservative margin of conservatism - a haircut or overlay reflecting the unquantified uncertainty - and a documented rationale. The bank should also treat vendor concentration and opacity as a separate risk, and set a contractual deadline: future renewals must include validation access, audit rights, and change-notification. If the vendor still refuses, the model should be downgraded to a restricted-use status or replaced. The key governance point is that the bank owns the risk decision even when it does not own the code.