The Reproducibility Dispute in a Psychology Lab
A psychology laboratory runs a replication of a published priming experiment. The original study reported a mean effect of 0.42 with p = 0.03 on a sample of 60 participants. The lab runs 240 participants and obtains a mean effect of 0.05 with p = 0.61. The lead researcher argues: 'A non-significant result proves the original effect does not exist, so we should publish a refutation.' A statistician on the team objects that the replication was underpowered for detecting effects smaller than the original estimate and that p-values do not measure the probability that a hypothesis is true. A third member suggests combining both datasets in a meta-analysis and reporting a confidence interval rather than a binary verdict. The lab must decide what claim, if any, its data license and how to frame the conclusion for publication.
1. Identify the methodological error in the lead researcher's inference and explain, using concepts from the methodology of science, why a non-significant result does not establish the absence of an effect.
Read the worked answer
The lead researcher commits the error of treating failure to reject the null hypothesis as acceptance of the null. In frequentist logic, a non-significant p-value means the observed data are not unlikely enough under the null to warrant rejection; it says nothing about the probability that the null is true. With 240 participants and an observed effect of 0.05, the confidence interval may still include small-to-moderate effects, so the data are compatible with a real but weaker effect.
A second error is conflating statistical significance with substantive existence. The original estimate of 0.42 carries substantial uncertainty; replication targets are often inflated by publication bias and winner's curse. A proper inference asks what range of effect sizes the new data exclude, not whether a threshold was crossed.
Finally, the claim 'the effect does not exist' is a universal negative that no finite sample can establish. The correct methodological conclusion is evidential: the replication provides evidence against an effect as large as the original, but not against all nonzero effects. This is why reporting effect sizes and intervals, not verdicts, is the scientifically defensible move.
2. Evaluate the third member's proposal to pool the datasets in a meta-analysis. What epistemic and methodological considerations should govern whether pooling is legitimate here?
Read the worked answer
Pooling is legitimate only if the two studies estimate the same parameter under sufficiently similar conditions. The lab must check whether procedures, stimuli, populations, and timing match. If the replication altered materials or context, the studies are not exchangeable and a pooled estimate would average heterogeneous quantities, producing a precise but meaningless number.
If the designs are comparable, a random-effects meta-analysis with a confidence interval is preferable to a binary verdict because it propagates uncertainty and quantifies heterogeneity. The lab should report the pooled estimate, its interval, and a measure of between-study variance, and interpret the result cautiously given only two studies.
Epistemically, pooling does not manufacture evidence for existence; it redistributes weight. A pooled estimate near zero with a wide interval still underdetermines the question. The defensible publication claim is conditional: 'Across these two samples, the evidence is compatible with effects up to X and inconsistent with the original large estimate.' This respects the asymmetry between confirming and disconfirming evidence and avoids overclaiming from a single replication.
3. The lab must choose between publishing a refutation, a null result, or a meta-analytic estimate. Recommend a course of action and justify it with reference to the norms of scientific methodology.
Read the worked answer
The recommendation should be to publish a transparent replication report centered on effect sizes and confidence intervals, framed as evidence about the magnitude of the effect rather than as proof of its absence, and to include the meta-analytic estimate only if design comparability is documented.
The justification rests on several methodological norms. First, the norm of total evidence: report all relevant data, including the original study, without selecting the framing that yields the cleanest narrative. Second, the norm of calibration: claims should match the strength of the evidence, so a wide interval warrants a modest claim. Third, the norm of falsifiability: the report should specify in advance what pattern would have counted against the hypothesis, which the pre-registered replication design can supply.
A pure refutation overclaims; a bare null result invites the same misinterpretation; a pooled estimate without comparability checks is spurious precision. The best course combines a pre-specified analysis, interval reporting, and explicit discussion of heterogeneity. This treats the replication as a contribution to a cumulative literature rather than a verdict, which is what the methodology of science actually licenses.