Credit-Limit Model: Fairness Drift, Risk Appetite and the Limits of AUC
Meridian Bank’s model risk committee reviews the “LimitUp” model, which recommends credit-limit increases for 1.2 million card customers. Validation shows AUC 0.81, stable for eight quarters. But approval rate is 64% for customers in high-income ZIP codes and 47% for low-income ZIP codes; the disparate impact ratio is 0.73 against a board risk-appetite threshold of 0.80. Population stability index is 0.26 over six months (threshold: 0.10 warning, 0.25 high). Complaints citing “unexplained limit cuts” rose from 120 to 410 per month. The model uses ZIP code, average balance, and payment history. A vendor supplies the feature engineering; the bank owns the scorecard. The CRO wants a recommendation in five days; the fair-lending team wants the model switched off; the business wants a challenger model in 90 days.
1. Is this a model risk issue or a compliance risk issue? Explain how the committee should treat both.
Read the worked answer
This is both a model risk and a compliance risk issue, and treating it as only one would miss the point. Model risk: the scorecard’s conceptual soundness is questionable if ZIP code is acting as a proxy; PSI 0.26 signals a material population shift; complaint growth suggests the model’s decisions are not being explained or are wrong for some customers. Compliance risk: a disparate impact ratio of 0.73 against a 0.80 appetite threshold is a breach of the bank’s own fair-lending standard; ZIP code can proxy for protected class; adverse action notices and UDAAP/consumer-protection duties are engaged. The committee should record both inherent risks, not net them. It should also distinguish a statistical disparity from a legal finding: a DIR below 0.80 is not automatically unlawful, but it is a red flag that requires fair-lending review and possible remediation. The CRO’s five-day deadline should produce an interim decision: freeze new limit increases in the affected segment or require manual review, preserve model documentation, and commission independent validation. Switching the model off entirely avoids new harm but may itself create customer and operational risk; the committee should compare options explicitly. The decision should be minuted with the risk appetite breach, the legal advice sought, and the residual risk accepted.
2. Recommend immediate and medium-term controls, and explain the residual risk that remains after each.
Read the worked answer
Immediate controls (0–30 days): stop using ZIP code as a feature and retrain or recalibrate; place a manual hold on limit increases for customers in the affected low-income segment until DIR is re-measured; issue accurate adverse action reasons; stand up a complaint triage with a 48-hour call-back; preserve versioned data and model artefacts; ask the vendor for feature engineering documentation and cooperation. Medium-term (30–90 days): independent fair-lending validation, challenger model without proxy features, explainability review, vendor contract review, and board reporting. Residual risk rationale: removing ZIP code may reduce proxy discrimination but does not guarantee DIR above 0.80 because other features may correlate with protected class. Manual review lowers immediate harm but introduces inconsistency and capacity risk. A challenger model in 90 days is realistic only if data are ready; the committee should set a go-live gate of DIR ≥0.85 in validation and PSI <0.10, not just AUC. Until then, residual risk remains high and should be reported as such. The committee should not accept the business case as a reason to keep the current model unchanged; it can accept a time-bound remediation plan with a hard stop.
3. Design the monitoring and escalation triggers. Why did stable AUC fail to surface this problem?
Read the worked answer
Monitoring should be layered. Monthly: DIR by protected segment and by income band, approval-rate gap, override rate, complaint rate per 10,000 accounts, adverse-action reason code distribution, and PSI. Weekly during remediation: manual-review queue ageing and overturned decisions. Triggers: DIR <0.85 warning, <0.80 halt new increases and escalate to the fair-lending committee; PSI >0.10 investigate, >0.25 revalidate; complaints >200/month or a 50% month-on-month rise escalate; override rate >15% suggests the model is misaligned with front-line judgement. Each trigger needs an owner, a response time, and a documented decision. AUC stability did not catch this because AUC measures overall rank-ordering, not subgroup calibration or the relationship between features and outcomes. A model can rank well overall while systematically shifting scores for one group; aggregate performance masks segment-level drift. Also, the approval-rate gap and complaint data are outcome indicators, not model-performance metrics. The lesson is to pair discriminatory-performance metrics with fairness, stability and operational-outcome indicators, and to review them together at the model risk committee.