Fairness Intervention That Would Hurt the Metric

Disparity in false-positive rates is a threshold and calibration question, and the answer depends on which fairness criterion the policy commits to.

The relevant criterion here is equalized odds (equal FPR and TPR across groups), because the harm described is being wrongly flagged for review - a burden, not a denial. Demographic parity would be the wrong lens: if base rates genuinely differ, forcing equal flag rates would misallocate scrutiny. So the business owner is partly right that base rates can differ, but that does not license unequal error rates. Equalized odds explicitly allows different base rates while requiring the model to be equally accurate about who actually defaults in each group.

What the data must show before concluding:

• Is the FPR gap driven by miscalibration or by a real signal gap? Check calibration curves per age band. If predicted probabilities are inflated for over-60s, a group-specific recalibration fixes FPR without dropping features.
• Is age a proxy? Age may stand in for retirement income patterns, credit history length, or thin files. If the true driver is credit-history length, the disparity is a feature-quality problem: the model is less informed about this group, so its errors are noisier. That changes the remedy.
• Sample size and confidence. 9% of applicants - is the 0.22 FPR estimated on enough defaults to be stable? Compute confidence intervals; a wide interval changes urgency, not direction.
• Label validity. Are defaults measured over a window that disadvantages this group (e.g., fixed 12-month window on a group with different repayment cadence)?

Conclusion: the disparity is a legitimate finding requiring explanation, not dismissal. The business owner's base-rate argument explains differing flag rates, not differing error rates; the latter needs a calibration or feature-quality explanation, or it is unfair under the policy's own criterion.

Removing age is the weakest option and likely counterproductive:

• AUC falls 0.86 → 0.81, so the model gets worse for everyone.
• FPR rises for under-40s - the fix redistributes harm rather than reducing it.
• Age is almost certainly recoverable from correlated features (credit history length, retirement income, account age), so removal provides no real protection while degrading accuracy. This is the classic fairness-through-unawareness trap.

Better alternatives, roughly in order of preference:

1. Group-specific threshold or recalibration. Keep the model, adjust the decision threshold per age band so FPR equalizes at 0.09. Cost: some loss in overall precision, and it requires legally defensible justification for treating groups differently. This directly targets the metric that is unfair.
2. Fix the underlying feature gap. If thin credit files drive the disparity, add alternative data or a model variant for thin-file applicants. This can raise AUC while closing the gap - the rare win-win. Requires data work and validation.
3. Retrain with a fairness constraint (equalized-odds regularization or post-processing). Keeps age as a feature but penalizes disparity. Trades some AUC for measured fairness; tune the trade-off explicitly with the business owner.
4. Human-in-the-loop for the over-60 band. Route flagged over-60 applicants to manual review before any adverse action. Reduces the impact of false positives without changing the model, at operational cost.

Trade-off framing for the business owner: quantify each option as (AUC change, FPR gap, approval-rate change, review volume). Option 1 is cheapest to implement; option 2 is the only one that may improve accuracy. Removal is dominated on every axis and should be rejected with the proxy argument stated explicitly.

Governance first, because the ambiguity is the risk.

• Resolve the conflict between the responsible-AI policy (age is sensitive) and legal's non-classification. Until resolved, apply the stricter standard: monitor and report age disparities. Document the decision, the owner, and the review date. An undocumented gap is the finding an auditor writes up.
• Record the chosen fairness criterion (equalized odds on FPR), the tolerance (e.g., gap ≤ 0.03), and the approved mitigation in a model card that ships with the model version.

Monitoring:

• Continuous disparity metrics on production decisions: FPR and TPR by age band, with confidence intervals, computed on labeled outcomes as they mature (defaults take months - so also monitor flag rates and score distributions in near real time as leading indicators).
• Calibration monitoring per group, since miscalibration is the most likely mechanism.
• Drift alerts on any feature that correlates with age; a proxy can emerge after a retrain even if age was removed.
• Alert thresholds tied to the tolerance, routed to a named owner with a response SLA, not a dashboard nobody reads.

Process controls:

• Fairness evaluation as a promotion gate in the registry: no model version reaches production without disparity metrics on the defined segments, reviewed by someone other than the author.
• Adverse-action explainability: for every flagged applicant, retain the top contributing features so a declined applicant can be given a reason - this is often a legal requirement independent of protected-attribute status.
• Periodic re-audit (quarterly) and a re-review trigger whenever the segment definition, label window, or feature set changes.
• Keep the audit finding, the analysis, and the decision in the model's registry record so the next reviewer sees the full history rather than rediscovering the gap.