Interpreting a p-value with a small sample
A school trials a new study programme on 12 students. Their mean improvement score is 4.2 points with sample standard deviation 5.0 points. The null hypothesis is that the population mean improvement is 0; the alternative is two-sided. A teacher runs a one-sample t-test and obtains t = 4.2 / (5.0/√12) ≈ 2.91 with 11 degrees of freedom, and a p-value of approximately 0.014. The teacher declares the programme 'proven effective at the 5% level.' A statistician colleague raises three concerns: the sample is small, the test assumes approximate normality of the underlying scores, and 'proven' is too strong a word. You must evaluate the test and the claim.
1. Verify the test statistic and p-value, state the decision at the 5% significance level, and interpret the p-value correctly.
Read the worked answer
Standard error = s/√n = 5.0/√12 ≈ 5.0/3.464 ≈ 1.443. Then t = 4.2/1.443 ≈ 2.91, matching the teacher's value. With df = 11, the two-sided p-value is about 0.014 (since P(T₁₁ > 2.91) ≈ 0.007, doubled).
At α = 0.05, since 0.014 < 0.05, we reject H₀. There is statistically significant evidence that the population mean improvement differs from 0.
Correct interpretation of the p-value: if the true mean improvement were 0, the probability of observing a sample mean at least as extreme as 4.2 (in either direction) is about 0.014 - about 1.4%. It is not the probability that H₀ is true, nor the probability the programme works. 'Proven effective' overstates the conclusion: rejecting H₀ is evidence against a zero effect, not proof of effectiveness, and a two-sided test does not by itself establish the direction or the practical size of the benefit.
2. Explain how the small sample size and the normality assumption affect the reliability of the conclusion, and suggest one design change that would strengthen the evidence.
Read the worked answer
With n = 12 the t-test relies on the sample mean being approximately normally distributed. The t-distribution handles the estimated standard deviation, but it assumes the underlying scores are roughly normal (or at least not strongly skewed or heavy-tailed). With only 12 observations, the central limit theorem gives little protection; a single outlier or strong skew could distort both the mean and the standard deviation, making the p-value unreliable. The sample standard deviation of 5.0 is itself estimated from 12 points, so it carries substantial uncertainty.
A design change that would strengthen the evidence: increase the sample size (which both improves the normality approximation and reduces the standard error), or use a paired design comparing each student before and after the programme to control for individual differences. A larger, paired, pre-registered study with a pre-specified effect size of practical interest would give a far more defensible conclusion than declaring the programme 'proven' from 12 students.