Deploying a Model Card When Subgroup Performance Is Uneven
A lending model predicts default probability to prioritize manual underwriting review. Overall validation AUC is 0.81, but disaggregated results show AUC 0.83 for the majority group and 0.74 for a smaller demographic subgroup that is 6% of applicants. At the chosen threshold, the subgroup has a false-negative rate of 0.38 versus 0.22 overall, meaning higher-risk applicants in that subgroup are more often passed to auto-approval. The subgroup has only 1,900 labeled examples. Legal asks whether the model can ship; the data team notes the subgroup estimate has wide confidence intervals and that the subgroup is historically under-served, so its labels may reflect past lending patterns rather than true risk.
1. Interpret the subgroup gap responsibly, distinguishing what the data can and cannot support.
Read the worked answer
The gap (AUC 0.74 vs 0.83; FNR 0.38 vs 0.22) is a real signal in the observed data, but with only 1,900 examples the confidence interval on subgroup AUC is wide - roughly, AUC standard error scales with 1/sqrt(n), so the subgroup estimate is far noisier than the overall one. The team should report CIs and not over-read a precise gap.
More importantly, the labels may be biased: if the subgroup was historically denied credit, its observed defaults reflect a selected population, not the true risk of all applicants in that group. A model trained on such labels can inherit and amplify the disparity. So the honest statement is: 'there is an apparent performance gap consistent with label bias and small-sample noise; we cannot yet separate the two.' That is different from 'the model is unbiased' and from 'the model is provably discriminatory.'
2. What concrete steps should precede a ship/no-ship decision?
Read the worked answer
(1) Quantify uncertainty: report subgroup AUC and FNR with confidence intervals and, if feasible, bootstrap them. (2) Audit the label mechanism: examine whether approval/denial history conditions who has observed outcomes; consider reweighting or a holdout of previously auto-approved applicants to get less biased labels. (3) Evaluate the decision, not just the score: compute approval/denial rates and error rates by group at the actual threshold, since FNR disparity is what causes harm here. (4) Consider mitigations: group-specific thresholds (with legal review), recalibration, or routing borderline subgroup cases to human review instead of auto-approval. (5) Document all of this in a model card with the subgroup caveats.
Only after these should legal decide. Shipping on overall AUC alone would hide a decision-relevant disparity.
3. Legal proposes simply removing the demographic attribute from the features. Evaluate this remedy.
Read the worked answer
Removing the protected attribute is insufficient and can be counterproductive. Correlated proxies (zip code, income patterns, credit history shape) can reconstruct group membership, so the model may still discriminate - 'fairness through unawareness' is a known fallacy. Worse, dropping the attribute removes the ability to measure subgroup performance, making the disparity invisible while it persists.
A better approach keeps the attribute out of the scoring features but retains it for auditing and mitigation. Then choose an explicit fairness criterion (e.g., equal false-negative rates, or equal approval rates) appropriate to lending law, measure against it, and apply a mitigation with documented trade-offs. Removing the variable is a gesture; measuring and constraining the outcome is the actual control.