Bias, Fairness, and Equity in Data
Bias enters data systems at every stage: historical/societal bias baked into labels, sampling and selection bias, measurement bias (proxies that misrecord reality), label bias, and deployment bias (misuse of a model in a new context). Candidates must know the fairness metrics and, crucially, that they cannot all hold at once. Key definitions include demographic parity (equal positive rates across groups), equalized odds (equal true- and false-positive rates), equal opportunity (equal true-positive rates), predictive parity (equal precision), and calibration. The impossibility result says that when base rates differ, these metrics generally conflict - so fairness requires an explicit, justified choice.
Traps: (1) assuming a "fairness metric" is universally correct; the right one depends on context and values; (2) confusing disparate treatment (different treatment) with disparate impact (neutral rule with unequal effect); (3) thinking removing a protected attribute removes bias - proxies (ZIP code, language) reintroduce it; (4) ignoring feedback loops where biased predictions shape future training data; (5) mistaking equality (same treatment) for equity (fair outcomes given different needs).
Rule of thumb: define the fairness objective before modeling, document the chosen metric and its trade-offs, and audit both inputs (proxies) and outputs (group-level error rates).
Worked example: A hiring model has equal selection rates across groups (demographic parity) but higher false-negative rates for one group. Parity hides the harm; equal opportunity analysis reveals it. The defensible response is to surface the trade-off and choose a metric aligned with the stated goal.