Reporting a Gender-Proxy Feature in a Hiring Algorithm
TalentSift sells an AI candidate-screening tool used by 300 employers. A data scientist discovers that the model's career-continuity feature penalizes candidates with employment gaps longer than six months. In the applicant pool, 68% of women have such gaps, compared with 22% of men, largely due to caregiving. The feature improves the model's predicted performance by 2.3 percentage points. The largest client, representing $3.2 million in annual revenue, is renewing in 30 days and has asked for the most predictive model. The data scientist raises the issue with her manager, who says employment gaps are not a protected class and the model is gender-blind. She escalates to the VP of product, who tells her to drop it. The company's code of conduct requires reporting suspected discrimination to the General Counsel; she does, and the General Counsel says there is no legal violation because the feature is facially neutral. She is considering external disclosure. She has documented the analysis.
1. Assess whether the career-continuity feature creates unlawful disparate impact and an ethical problem, and evaluate the General Counsel's conclusion.
Read the worked answer
The feature is facially neutral but produces a substantial disparity: 68% of women versus 22% of men have gaps longer than six months. Under Title VII disparate-impact doctrine, a plaintiff must identify a specific practice, show a statistically significant disparity, and then the employer must show job-relatedness and consistency with business necessity; a less discriminatory alternative can defeat that defense. The feature improves predicted performance by only 2.3 percentage points, which is weak evidence of necessity and does not address whether a less discriminatory alternative exists. The General Counsel's statement that there is no violation because the feature is facially neutral is incomplete: disparate-impact liability does not require intent or a protected-class variable. The GC may be giving a risk assessment rather than a legal conclusion, but the company should not treat it as the final word. Ethically, penalizing caregiving gaps can entrench gender inequality and is difficult for candidates to understand or contest. The client's demand for the most predictive model does not override fair employment obligations.
2. Apply a whistleblowing decision framework to determine whether external disclosure is justified at this stage. Identify the conditions and alternatives.
Read the worked answer
A useful framework asks whether the concern involves serious harm, whether the purpose is proper, whether internal channels are exhausted, whether the disclosure is proportionate, and whether there is a reasonable chance of correction. Here, the harm is serious and widespread: 300 employers may use a tool that disadvantages women. The purpose is proper, not personal gain. But internal channels are not exhausted. She has reported to her manager, the VP, and the GC, but the company may have an ethics hotline, a board audit committee, or a compliance committee. She should escalate in writing to the board or audit committee and ask for a documented response. A regulatory complaint to an equal employment opportunity agency may be a protected and more proportionate step than media disclosure. External disclosure may become justified if the feature is about to be shipped or renewed without change, if retaliation occurs, or if the board refuses to act. At this stage, external disclosure is premature. She should preserve her documentation and seek independent legal advice about whistleblower protections.
3. What protections and documentation should she secure, and what should the company do to remediate?
Read the worked answer
She should keep dated copies of her analysis, emails, meeting notes, and the model evidence in a safe location; confirm the company's whistleblower and anti-retaliation policy; and seek independent legal advice about protected activity. She should communicate in writing and ask for written responses. If she files a regulatory complaint, she should do so within the applicable deadlines and keep the complaint confidential where required.
The company should remove or transform the career-continuity feature, retrain and validate the model, and test for proxy effects. It should notify affected clients, provide guidance on lawful use, and audit outcomes by gender and caregiving status. It should also review its code of conduct and escalation process so that a GC opinion is not treated as the end of the discussion. Governance should include a model risk committee, a documented dissent process, and anti-retaliation training. Finally, the company should consider whether candidates and clients deserve transparency about the change. Remediation is not only a technical fix; it is a test of whether the organization can hear an internal objection and act on it.