The Replication That Would Not Repeat
A Year 12 student reads a 2019 paper reporting that students who wrote summaries in blue ink scored 12% higher on a recall test than those writing in black ink (n = 40, p = 0.03). She decides to replicate it for her Science Extension research project. She recruits 60 classmates, randomly assigns them to blue or black pens, and finds a mean difference of 1.4% in the opposite direction (p = 0.61). Her supervisor says the original finding was probably just noise, but the student argues that her sample was larger, so her null result must be the more reliable one and the original study should be considered disproven. She wants to title her report 'Blue Ink Does Not Improve Memory: A Refutation'.
1. Evaluate the student's reasoning that her larger sample makes her null result definitive and the original finding 'disproven'.
Read the worked answer
The student is conflating failure to replicate with refutation. A non-significant result (p = 0.61) means the data are compatible with no effect, but it does not establish that the effect is zero - it establishes that her study lacked evidence for it.
Her logic has three problems:
- A null result is not evidence of absence. With n = 60 split across two groups, her study has limited power to detect a small effect. If the true effect is genuinely small (~2%), her design may have under 30% power to detect it, so a null result is the expected outcome even if the effect is real.
- Larger n improves precision, not truth. A bigger sample narrows the confidence interval around her estimate; it does not make her estimate the correct one. The two studies may simply be sampling from the same population with different noise.
- Publication and design differences matter. The original study's effect could depend on context (age, task, timing, ink shade), so a failed replication may reflect a different operationalisation rather than a false original.
The honest conclusion is that the evidence is inconsistent and inconclusive, and the effect is likely small or context-dependent. The title 'A Refutation' overstates what the data support.
2. Recommend two specific changes to strengthen the study's inferential value, and justify each in terms of scientific reasoning.
Read the worked answer
Change 1 - Increase power and pre-specify an effect size of interest. The student should conduct an a priori power analysis targeting the smallest effect she cares about (e.g. 5%). If that requires ~200 participants per group, she should either recruit more or explicitly reframe the study as a close replication with a stated minimum detectable effect. Justification: without adequate power, a null result is uninformative, and pre-specifying the effect size prevents post hoc rationalisation.
Change 2 - Control and report potential confounds and use equivalence testing. She should standardise ink colour (measured with a colourimeter), time of day, and testing environment, and report them. She should also analyse the result as an equivalence test - asking whether the observed difference falls within a pre-set equivalence bound - rather than treating p > 0.05 as proof of no effect. Justification: equivalence testing directly addresses the question 'is the effect small enough to be practically negligible?', which is what her title claims, and controlling confounds rules out alternative explanations for a null result.
A third option would be a meta-analytic combination of her data with the original, but this requires access to the original raw data and should be framed cautiously.