AI Triage in a Regulated Lending Pipeline
A mid-sized lender deploys a third-party AI model to triage loan applications, flagging 'high-risk' files for manual review. The model is trained on five years of historical decisions. Within three months, internal audit finds the flag rate for applicants in two ZIP codes is 40% higher than the portfolio average, and the vendor will not disclose the model's feature weights, citing trade secrets. A regulator's fair-lending examination is scheduled in six weeks. The vendor's contract disclaims all warranties and places all compliance responsibility on the lender. The lender's model risk policy requires explainability for 'material' models but has no definition of material and no AI-specific approval gate.
1. What is the lender's exposure, and who owns the compliance obligation?
Read the worked answer
The lender owns it. Outsourcing the model does not outsource the obligation - fair-lending and consumer-protection duties attach to the lender's decisions regardless of who built the tool.
Exposure:
- Disparate impact: a materially higher flag rate in specific ZIP codes is a classic red flag for discriminatory effect, even if ZIP code is a proxy rather than a protected characteristic.
- Explainability gap: the vendor's trade-secret refusal collides with the lender's own policy and with examiner expectations that the lender can explain adverse decisions.
- Vendor contract: a warranty disclaimer and a compliance-shifting clause are unenforceable against the regulator. They allocate risk between the parties; they do not reduce the lender's liability.
- Governance gap: an undefined 'material' threshold means the explainability policy was never triggered.
This is a model risk governance failure as much as a fairness problem.
2. What should the lender do before the examination, and what durable controls should follow?
Read the worked answer
Before the exam (six weeks):
- Quantify the disparity - measure flag rates by protected class proxies, not just ZIP code, and document methodology.
- Suspend or shadow-run the model for the affected segments while maintaining manual review; do not simply disable triage and create a backlog without a plan.
- Demand explainability from the vendor as a contract-remediation condition; obtain at least feature-importance documentation or a model card.
- Prepare a candid remediation narrative for the examiner - examiners penalize concealment more than the underlying finding.
Durable controls:
- Define 'material' model with objective thresholds (decision volume, consumer impact).
- AI approval gate before deployment: fairness testing, explainability requirement, and documented owner.
- Contract terms requiring model documentation, audit rights, and change notification.
- Ongoing bias monitoring with defined escalation triggers.
The judgment: the vendor contract allocates cost, not accountability; and the fastest path to a worse examination outcome is an undisclosed, unquantified disparity.