The Forecasting Model That Learned the Wrong Thing
A bank builds a machine learning model to predict which small-business loan applications should be fast-tracked for approval. The training data covers five years of past decisions. The model performs well (AUC 0.86) and the bank plans to deploy it to reduce approval times. A review finds: (a) two of the strongest predictors are the applicant's postcode and the branch that processed the application; (b) historical approval rates for applicants from three lower-income postcodes were 40% lower than the average, controlling for the same financial metrics; (c) the model's training labels are the human decisions, not actual loan performance; (d) the bank's policy requires that credit decisions be explainable to the applicant on request.
1. Identify the governance and ethical risks in this model and explain why each matters under the bank's explainability policy.
Read the worked answer
Four risks, each material:
- Proxy discrimination. Postcode and branch are not creditworthiness. If approval rates for three lower-income postcodes were 40% lower after controlling for financial metrics, postcode is encoding something other than risk - likely historical bias or branch-level discretion. Using it as a predictor reproduces and amplifies that bias at scale. This is a fairness and potentially legal risk, and it is not explainable to an applicant in any defensible way ('you were declined because of your postcode' is not a credit reason).
- Label leakage from human decisions. The model is trained to imitate past human choices, not to predict loan performance. It will inherit every bias and inconsistency in those choices and will not improve on them. Its AUC of 0.86 measures agreement with humans, not predictive validity. This is the root cause of risk 1.
- Explainability failure. The policy requires reasons on request. A model relying on postcode and branch cannot produce a reason that satisfies the policy or the applicant. A post-hoc explanation that says 'postcode contributed 0.3' is technically true and substantively useless.
- Accountability gap. No named owner for the model's decisions, no documented validation of fairness, no monitoring plan. When an applicant challenges a decision, no one can reconstruct the basis.
The AUC is a distraction: high accuracy on biased labels is not evidence of a good model.
2. Recommend a remediation path that would make the model deployable, and state what must be true before go-live.
Read the worked answer
Remediation:
- Re-label. Train on actual loan performance (default, delinquency, loss) rather than human decisions. This is the single most important change; without it the model is a bias-copying machine.
- Remove or justify proxy features. Drop postcode and branch. If geography carries genuine risk signal, replace it with a documented, tested risk factor and demonstrate it does not proxy for protected characteristics. Run fairness testing (e.g. approval-rate parity and error-rate parity across postcode groups) before and after.
- Build explainability in. Use a model class or explanation method that produces applicant-facing reasons aligned to the policy. Test the explanations with real cases, including declined applicants.
- Governance. Name a model owner, document the validation, set up ongoing monitoring for drift and fairness, and define a challenge/appeal route.
Before go-live, all of the following must be true:
- The model predicts performance, not past decisions, and has been validated out-of-sample.
- Fairness testing shows no unjustified disparity, or disparities are documented and accepted by an accountable committee.
- Every decline can generate a policy-compliant reason.
- A monitoring and review cadence is in place with a named owner.
Until then, deploy only as a decision-support tool with human review, not as an automated fast-track.