Framing, Experimenter Sex, and a Mediation Claim in a Stereotype-Threat Study
Researchers recruited 120 undergraduates (60 women, 60 men) who had each completed one semester of calculus. Participants were randomly assigned to one of two conditions. In the ability-diagnostic condition, a male graduate student read a script saying the 20-item mental-rotation test measures spatial ability that predicts success in engineering. In the puzzle condition, a female graduate student read a script saying the same items were a set of puzzles being piloted. Women in the diagnostic condition scored 8.2 points lower than women in the puzzle condition (95% CI 1.4 to 15.0, p = 0.03), while men's scores differed by 0.6 points (p = 0.71). After the test, participants rated their anxiety from 1 to 7; women in the diagnostic condition reported more anxiety (4.8 versus 3.9, p = 0.04). The authors state that anxiety mediated the performance gap, conclude that stereotype threat explains it, and recommend that engineering admissions tests eliminate ability-diagnostic framing.
1. Identify the design features that prevent a mechanistic conclusion, and describe the minimal design that would isolate stereotype activation.
Read the worked answer
Condition is completely confounded with experimenter sex, and the two scripts differ in at least three ways: diagnostic versus pilot framing, a claim that the test predicts engineering success, and the person delivering the message. Any of these could produce an 8-point difference, so the study cannot attribute the effect to stereotype activation. A same-sex experimenter effect is a live alternative: women may underperform with a male experimenter regardless of framing. The minimal fix is a fully crossed 2 × 2 design (framing × experimenter sex) with a verbatim script, identical timing and item order, and condition-blind scoring. Add a manipulation check asking what participants were told the test measures, and add an evaluative-but-not-stereotype-relevant framing, such as predicting success in a non-technical field, to separate general evaluation apprehension from stereotype threat. Pre-register the primary contrast so the key test is not chosen after seeing the data.
2. Evaluate the mediation claim. What evidence would be needed to support it, and what does the current result actually show?
Read the worked answer
Mediation requires temporal order and a measured mediator on the causal path: framing leads to anxiety, which leads to performance. Anxiety here was measured after the test, so it may be a consequence of performing poorly or of appraising the items as difficult rather than a cause of the errors. The design cannot distinguish anxiety causing errors from errors causing anxiety. Even with correct timing, a single self-report item is a coarse proxy, and p = 0.04 on one of several comparisons needs correction for multiplicity. Proper support would require measuring anxiety and a physiological index such as heart rate before and during the test, estimating the indirect effect with bootstrapped confidence intervals, and, ideally, manipulating anxiety independently, for example with a brief reappraisal instruction, to show that changing it changes performance. As reported, the evidence supports correlated with, not mediated.
3. Interpret the 8.2-point difference, the p-value, and the confidence interval. Is the policy recommendation supported by this study?
Read the worked answer
The 8.2-point difference is between two groups of women, not a male–female gap, so it cannot by itself explain a sex difference in engineering participation. The p-value of 0.03 means that, assuming no true effect and all model assumptions, data this extreme would arise about 3% of the time; it is not the probability that the hypothesis is true and it says nothing about effect size. The 95% CI of 1.4 to 15.0 is wide, so the true effect could be trivial or very large. With 60 women split across conditions, the study is underpowered for a precise estimate and for testing moderation. The recommendation may still be defensible on other grounds, since removing irrelevant framing is low-cost, but this study supports a narrower claim: a brief, multiply confounded manipulation changed scores. Replication with the crossed design and a pre-registered analysis is the minimum before citing a mechanism.