The coding test and the interview
A tech firm finds that 18% of women and 34% of men pass its timed algorithmic screening test. HR introduces a blind resume review, which raises the share of women invited to test from 21% to 30% - but the pass rate on the test does not move. In a pilot, the same test was administered to equivalent groups of women under two framings: one described it as 'a validated measure of programming aptitude,' the other as 'a set of logic puzzles we're piloting.' Pass rates for women were 19% under the first framing and 31% under the second; men's rates were 33% and 34%. Interviewers who see a passing score then rate candidates; on average, women who pass receive lower 'leadership potential' ratings than men with identical scores.
1. Diagnose the two distinct problems the firm faces, and identify what evidence separates them.
Read the worked answer
Two different mechanisms are operating and they need different fixes.
Problem 1 - performance under evaluative stereotype. The pilot is the key evidence: only the framing changed, and only women's pass rates moved (19% → 31%, an 12-point swing), while men's barely budged (33% → 34%). A difference produced by a label alone, with no change in test content or ability, is the signature of stereotype threat - evaluative apprehension that consumes working-memory resources when a negative group stereotype is made salient by an 'aptitude diagnostic' framing. The interaction (effect only in the stereotyped group) is what separates threat from a general test-anxiety or difficulty effect.
Problem 2 - biased evaluation of identical credentials. Women and men with the same passing score get different leadership ratings. Because the score is held constant, this cannot be a performance difference; it is a judgment difference. That is better described by shifting standards / status characteristics reasoning (the same evidence is weighed against a higher implicit bar) than by stereotype threat, which acts on the performer, not the rater.
The blind-resume result confirms a third, upstream problem: screening was biased, but fixing screening cannot fix either downstream mechanism - hence no movement in pass rates.
2. Which single change should the firm prioritize, and what is the strongest argument against your choice?
Read the worked answer
Prioritize the evaluation stage - structured, score-anchored rubrics for the leadership interview, raters who never see demographic information, and calibration on identical vignettes. Argument: the score-held-constant discrepancy means the firm is discarding qualified candidates after they have already demonstrated the skill, so the loss is direct and the intervention is cheap and measurable.
Strongest counter-argument: the test's pass-rate gap is larger in absolute terms (34% vs 18%), so fixing interviews while leaving a threat-inducing test in place still shrinks the pool the fair interview draws from. Also, the pilot framing manipulation was small and single-site; a label effect of that size may not replicate, in which case effort spent on framing is wasted while the rater bias is well documented within the firm's own data.
A defensible resolution: run the low-threat framing as a low-cost default immediately (it costs nothing to relabel), and treat rater training as the primary investment because it addresses a discrepancy the firm has already measured on its own raters.
3. A manager argues: 'We removed names from resumes, so bias is handled. The remaining gap must be a genuine ability difference.' Evaluate this reasoning.
Read the worked answer
It commits a disjunction fallacy: bias eliminated at one stage is treated as bias eliminated everywhere, so any residual gap is attributed to the person rather than the situation - a fundamental attribution error applied to a group.
The firm's own data contradict it on two counts. First, the pilot held ability constant in every respect that matters (equivalent groups, same problems) and still produced a 12-point gap in women's pass rates purely from a sentence of framing. Second, interviewers rated equally scoring candidates differently - a gap that cannot be ability, since ability was measured and matched.
There is also a measurement point: a 'validated aptitude' label does not guarantee the test is unbiased; validity for predicting job performance is a separate question from whether subgroup scores are affected by situational cues. So 'the test is validated' answers a different question than 'does this test measure the same thing in both groups under this framing.'