The Fraud Model That Flags One Neighborhood
A bank deploys a fraud-detection model that scores transactions in real time. The model uses 40 features, including merchant category, transaction amount, time of day, and a 'distance from home address' feature. After six months, an internal audit finds that customers in three low-income ZIP codes have a false-positive rate of 6.8%, versus 1.9% elsewhere. Those customers are disproportionately routed to manual review, which delays transactions by up to 48 hours and, in 12% of flagged cases, results in account restrictions. The model's overall AUC is 0.88 and has not changed. The data science team says the ZIP-code effect is not caused by a protected attribute because ZIP code is not a protected class and was never a model input. The compliance team notes the bank must explain adverse actions to customers. Customer complaints from the affected areas have tripled.
1. Evaluate the data science team's defense that ZIP code is not an input and therefore no discrimination issue exists.
Read the worked answer
The defense confuses input features with outcomes. Discrimination analysis asks whether a model produces disparate outcomes across groups, not whether a particular field appears in the feature list. A model can produce disparate impact through proxies: home address feeds the 'distance from home address' feature, and address is strongly correlated with neighborhood and therefore with the ZIP codes showing elevated false positives. Merchant category and time-of-day patterns can also correlate with geography and income.
The unchanged AUC of 0.88 is irrelevant to this question. AUC is a global ranking metric; it can stay flat while error rates diverge sharply across subgroups, because gains in one group offset losses in another. A 6.8% versus 1.9% false-positive rate is a 3.6x disparity, and the downstream consequences (48-hour delays, 12% account restrictions) are material harms, not inconveniences.
The team also needs to check whether the training labels are biased. If fraud investigations historically concentrated on certain areas, the labels encode enforcement patterns rather than fraud, and the model learns to reproduce them. That is a label-quality problem, not a feature-list problem. The right response is a subgroup error audit, proxy analysis on the address-derived features, and a review of how fraud labels were generated.
2. The bank must explain adverse actions. Describe what an explanation would need to contain and where the current model falls short.
Read the worked answer
An adverse-action explanation must tell the customer the specific principal reasons the decision went against them, in terms they can understand and, where relevant, act on. For a fraud model this is harder than for credit, because the action is often a hold or restriction rather than a denial, but the same logic applies: the customer is entitled to know why.
What the explanation needs:
- The principal factors driving the score for that transaction, not a generic list of all 40 features.
- Reasons stated at a level of specificity that is truthful and useful, e.g. 'transaction distance from your usual area' rather than 'risk model output.'
- A path to contest or resolve, since a 48-hour delay with a 12% restriction rate is a real burden.
Where the current model falls short: a 40-feature real-time model with no described explanation layer cannot produce per-decision reasons. Worse, if the explanation surfaces 'distance from home address,' it will reveal the geographic proxy that drives the disparity, which is both a fairness problem and a communications problem. The bank should not paper over this with vague language; it should fix the feature and then explain honestly. A workable path is to add a post-hoc attribution method, validate that attributions are stable and faithful, and route high-impact actions through human review with a documented reason.