The Model That Passed Validation and Should Not Ship
You lead a data science team at a consumer lender. Your team built a credit-scoring model to replace a rules-based system. On the held-out test set it performs well: AUC 0.81, and it clears the fairness threshold the compliance team set (the disparity ratio between demographic groups is 0.87, above the 0.80 minimum). The model uses 40 features, including postcode, employment tenure, and device type. A junior data scientist flags that device type is acting as a proxy - it correlates strongly with a protected characteristic and contributes 9% of the model's predictive power. Removing it drops AUC to 0.79 and the disparity ratio to 0.84. The business sponsor says the model passes the stated threshold and the proxy concern is "academic." Launch is in three weeks and the rules-based system is costing measurable revenue. The compliance sign-off is already obtained.
1. The model meets the stated threshold and has compliance sign-off. Do you ship it? Justify your decision.
Read the worked answer
Meeting the threshold and having sign-off are necessary conditions, not sufficient ones. I would not ship in the current form without escalating the proxy finding, and here is the reasoning.
Why "it passes the threshold" is not decisive:
- The threshold is a floor set in advance, not a certification that the model is fair. A disparity ratio of 0.87 that is produced by a proxy for a protected characteristic is a different thing from 0.87 produced by legitimate risk factors. The number alone cannot distinguish them, and the compliance sign-off was given without knowledge of the proxy finding.
- The proxy contributes 9% of predictive power. That is not marginal. It means a meaningful share of the model's decisions are being driven by a variable that stands in for a protected characteristic.
- "Academic" mischaracterizes it. Proxy discrimination is the central failure mode in credit scoring and the reason disparate-impact scrutiny exists.
What the finding changes:
- Removing device type costs 0.02 AUC and improves the disparity ratio to 0.84. The model without the proxy is both fairer and barely less accurate. That trade is close to free.
- The compliance sign-off was obtained on incomplete information. It needs to be revisited, not treated as settled.
My decision: do not ship as-is. Present the proxy analysis to compliance and the sponsor, and recommend shipping the version without device type. If the sponsor insists on the full model, escalate in writing with the analysis attached. Three weeks is enough to retrain and re-validate the reduced model; it is not enough to defend a known proxy in a lending decision.
2. The business sponsor argues that removing device type costs revenue and the disparity ratio is still above threshold either way. How do you make the case, and what do you do if you are overruled?
Read the worked answer
The sponsor's argument has a real economic premise and a flawed fairness premise. I address both on their own terms.
Making the case:
- Quantify the revenue at stake. A 0.02 AUC difference is small; translate it into expected approval-rate or loss-rate impact. Often the revenue argument shrinks when expressed concretely. If it is genuinely large, that is a fact to weigh, not to dismiss.
- Reframe the threshold. The threshold answers "is the disparity acceptable?" It does not answer "is this disparity caused by a protected-characteristic proxy?" A model can clear the first and fail the second. Compliance signed off on the first question only.
- Point at the asymmetry. The fairer model is 0.02 AUC worse and 0.03 better on disparity. The cost of removing the proxy is small; the cost of a disparate-impact finding in lending is large - remediation, refunds, regulatory action, reputational damage. The risk-adjusted comparison favors removal.
- Note the precedent. Shipping a known proxy sets the standard for every model that follows: pass the threshold, ignore the mechanism. That is a governance failure that compounds.
If I am overruled:
- Document the decision and my recommendation in writing, with the proxy analysis attached, and make sure compliance has seen the finding before launch. Their sign-off must be informed.
- Escalate through the appropriate channel - model risk, internal audit, or the DPO-equivalent for fairness - not around the sponsor but above or beside them.
- If the launch proceeds, ensure monitoring tracks outcomes by group post-launch so the proxy effect is measured in production, with a defined trigger for review.
- I do not sabotage or slow-walk the launch. I make the risk visible to the people accountable for it and let the accountable decision-maker decide with full information. That is the professional obligation; the decision authority is theirs, not mine.