Fairness, Bias, and Non-Discrimination in Data and Algorithms
Scope
Fairness items test whether you can locate bias in a pipeline, choose a defensible metric, and recognise when two fairness criteria cannot hold at once.
Where bias enters
- Historical bias: the world the data reflects is already unjust.
- Representation bias: sampling excludes groups.
- Measurement and label bias: proxies such as arrests for crime or spending for need encode enforcement or access patterns.
- Aggregation bias: one model applied to heterogeneous groups.
- Deployment bias: the tool is used outside the conditions it was validated for.
Metrics and their conflicts
- Demographic parity: equal positive rates across groups; blind to base rates.
- Equalized odds: equal true positive and false positive rates.
- Equal opportunity: equal true positive rates; a relaxation of equalized odds.
- Predictive parity: equal precision across groups. Chouldechova and Kleinberg and colleagues showed that with unequal base rates, calibration, equal false positive rates and equal false negative rates cannot all hold simultaneously. A fairness claim without a named metric and named group is therefore meaningless.
Law and doctrine
Disparate impact doctrine (Griggs v. Duke Power; the EEOC Uniform Guidelines four-fifths rule) reaches facially neutral practices with disproportionate effect. US credit (ECOA and Reg B), housing (Fair Housing Act) and employment (Title VII) each define their own protected classes and tests. EU law treats indirect discrimination similarly under the gender equality directives and the Race Equality Directive. Fairness through unawareness - deleting the protected attribute - fails because proxies carry the signal.
Worked example
A résumé screener is trained on ten years of hires who were mostly male. Removing the gender field does not help: names, pronouns and university proxies carry the signal. An audit shows equal precision but a much lower true positive rate for women. The team adopts equal opportunity as the operational target, adds disaggregated monitoring, audits the label itself because manager ratings are biased, and documents the trade-off. They do not claim the model is fair in general - only that specified metrics meet thresholds for specified groups.
Traps
Assuming protected attributes are the only route to bias; treating accuracy as fairness; choosing a metric after seeing the results; ignoring intersectional groups too small to audit confidently; forgetting that the label, not the algorithm, is often the defect.