The Poisoned Feedback Loop
A fintech deploys an AI model that scores loan applications. The model is retrained monthly on the prior month's approved applications and their repayment outcomes. In January the model began down-ranking applicants from two ZIP codes. Investigation shows that 14 months earlier those ZIP codes were flagged by a legacy rule and largely auto-declined by a human reviewer, so few of their applications were ever approved. The training pipeline only sees approved loans, so it learns that applicants from those ZIPs "don't appear," and the pattern compounds each cycle. A regulator has asked for an explanation of the model's disparate impact. The data science lead says the model is technically accurate on the data it was trained on and proposes simply adding more features to improve AUC.
1. Diagnose the mechanism producing the disparate impact and explain why "the model is accurate on its training data" is not a defense.
Read the worked answer
The mechanism is selection bias / a feedback loop from a restricted label set. The model is trained only on approved applications. Because an earlier human/legacy rule suppressed approvals in two ZIPs, those ZIPs are underrepresented in the training data. The model learns their near-absence as a signal, down-ranks them further, which suppresses future approvals and reinforces the gap - a self-fulfilling loop, not a discovered truth about creditworthiness.
"Accurate on its training data" is not a defense because accuracy is measured against a label distribution that the system itself helped create. The model can be highly accurate at predicting who would have been approved under the biased prior process while being wrong about actual repayment risk. The target variable is contaminated. Statistical fit does not establish that the model measures the construct (creditworthiness) rather than the historical decision policy.
This is a construct-validity and sampling problem, not a tuning problem.
2. The lead proposes adding features to raise AUC. Assess this proposal against the regulator's question.
Read the worked answer
Adding features to raise AUC does not address the regulator's question and may worsen the problem.
- The regulator asked for an explanation of disparate impact, not for higher predictive accuracy. AUC on a biased label set can rise while the disparity persists or grows; the metric is orthogonal to the harm.
- More features give the model more ways to reconstruct the ZIP-correlated pattern indirectly (proxies), making the disparity harder to detect and explain - the opposite of what an explanation requires.
- It signals the team is optimizing the wrong objective.
What the regulator needs: a documented account of how the training population was selected, evidence of whether the outcome disparity is justified by a legitimate, non-proxy factor, and a remediation plan. Feature engineering without changing the sampling/labeling or measuring impact by group does not answer that.
3. Propose the minimum remediation set the fintech should implement before responding to the regulator.
Read the worked answer
Minimum defensible remediation:
- Break the feedback loop in training data: include declined applications and their outcomes where lawfully observable, or use a sampling/reweighting scheme that does not condition the training set on the prior biased approval decision. Correct or exclude the contaminated label.
- Measure impact by group and by proxy: compute approval and error rates across the affected ZIPs and protected-class proxies, and test whether ZIP is acting as a proxy. Document results.
- Establish a fairness acceptance gate in the retraining pipeline so a model cannot be promoted if disparity exceeds a defined threshold - making this a control, not a one-time analysis.
- Human review and appeal path for down-ranked applicants, with logging, so individual harms are catchable.
- Produce the explanation artifact for the regulator: data lineage, the identified mechanism, the remediation, and ongoing monitoring commitments.
Adding features is optional and secondary; none of the above can be skipped.