The Fraud Model With a Fairness Complaint
A bank deploys a gradient-boosted fraud-detection model that flags transactions for manual review. The model uses features including transaction amount, merchant category, device age, IP geolocation, and 'account tenure.' An internal audit finds that the flag rate for customers in low-income ZIP codes is 2.3× the rate for high-income ZIP codes, even after controlling for actual fraud rate. The model's AUC is 0.91 overall. The data science team argues that ZIP code is not a protected attribute and that the model is 'just predicting fraud,' so no action is needed. Meanwhile, the manual-review queue is capacity-constrained: reviewers can handle 1,000 cases per day, and the model currently flags 1,400.
1. Assess the team's argument that no action is needed, addressing both the legal/ethical framing and the technical mechanism.
Read the worked answer
The argument is technically and ethically incomplete. Even if ZIP code is not a protected class, it is a proxy for race and socioeconomic status; disparate impact doctrine (in many jurisdictions) and fair-lending principles can apply regardless of whether the attribute is used directly. 'Just predicting fraud' is not a defense if the model's errors fall unevenly.
The technical mechanism matters. Features like device age, IP geolocation, and account tenure may be correlated with the protected group through historical access patterns (e.g., newer devices, shared IPs, shorter tenure among lower-income customers). The model may be learning a proxy for group membership rather than fraud risk. AUC 0.91 overall says nothing about subgroup calibration or error-rate parity: a model can have high AUC and still have much higher false-positive rates for one group.
The correct first step is measurement: compute false-positive and false-negative rates by group, calibration curves by group, and feature-importance/SHAP analysis to see which features drive the disparity. Then decide whether to remove, transform, or constrain those features, apply group-specific thresholds, or add fairness constraints during training. The team's conclusion should be 'we need to measure and mitigate,' not 'no action.'
2. Given the 1,000-case review capacity and 1,400 current flags, propose how to set the decision threshold and what fairness-aware operational policy to adopt.
Read the worked answer
Threshold setting: The current threshold is mis-calibrated to capacity. Set the operating point so expected flags ≤ 1,000/day, using the precision–recall curve and the cost of false negatives vs. false positives. Because fraud costs are asymmetric, choose the threshold that maximizes expected net benefit subject to the capacity constraint, and document the trade-off (e.g., 'we accept X additional missed frauds to stay within review capacity').
Fairness-aware policy: A single global threshold may preserve the disparity. Consider:
- Equalized odds / equal opportunity constraints (e.g., equal false-positive rates across groups) implemented via threshold adjustment or constrained training.
- Group-specific thresholds only if legally permissible and justified by measured error disparity; document rationale.
- Feature remediation: drop or transform proxy features (e.g., bucket device age, remove raw IP geolocation) and retest the disparity.
Operational safeguards: Route borderline cases to a lower-cost secondary screen rather than straight to manual review; monitor flag rates by group weekly; provide an appeals path for customers; log decisions for audit. Re-evaluate the model whenever the review capacity or fraud base rate changes, since the threshold is capacity-dependent.