The Cluster That Was Counted as a Crowd
With ethics approval, a university team recruits 12 Year 9 classes from six Auckland secondary schools to test a four-week mindfulness programme. Six classes are randomly assigned to the programme and six to a waitlist. The primary pre-registered outcome is the 10-item Perceived Stress Scale at eight weeks; a secondary outcome is school-record attendance. Of 240 consented students, 163 complete the eight-week assessment. The four programme teachers volunteered and received two days of training; waitlist teachers receive training after the study. The analysis ignores class membership: the intervention group scores 2.4 points lower on the PSS-10, d = 0.31, p = .04. Attendance is 3.1 percentage points higher in intervention classes, p = .21. The team's draft says the programme 'reduces stress and improves attendance'.
1. The p = .04 result is described as evidence of effectiveness. Explain the unit-of-analysis problem and what the team should report instead.
Read the worked answer
The p value treats each of the 163 students as an independent observation. They are not: students are nested in classes and classes in schools, so the effective sample size is closer to 12 classes than 163 students. With a typical intraclass correlation for school-based wellbeing outcomes of, say, .05–.10, the design effect is 1 + (m − 1)ICC. With about 27 students per class at follow-up, that is 1 + 26 × .05 = 2.3 to 1 + 26 × .10 = 3.6. The standard error is therefore roughly 1.5–1.9 times larger than reported. A d of 0.31 with p = .04 unadjusted would probably not survive cluster-robust standard errors or a multilevel model with school fixed effects and class-level random intercepts; the 95% confidence interval would likely include zero. The team should report the ICC, the design effect, the number of clusters, and a cluster-robust or multilevel estimate, preferably with a small-sample correction such as Kenward–Roger. With only 12 clusters they should state plainly that the trial is underpowered for cluster-level inference and present the result as hypothesis-generating, not confirmatory.
2. Evaluate the volunteer-teacher and waitlist features as threats to internal validity, and recommend feasible remedies.
Read the worked answer
Volunteer teachers are not a neutral delivery mechanism. They are likely to be more motivated, more confident with wellbeing content, and perhaps more skilled at classroom management than teachers who did not volunteer. If those qualities also affect student stress or attendance, the programme effect is confounded with teacher selection. The waitlist controls passage of time and repeated testing, but it does not control expectancy: programme teachers and students know they are receiving something, and attendance records are collected by staff who are not blind to condition. A waitlist also creates a no-treatment contrast, so the result cannot separate the mindfulness content from extra attention, novelty, or teacher enthusiasm. Feasible remedies include randomising teachers before allocation, measuring and reporting baseline equivalence on teacher characteristics, using an active control such as a matched relaxation or study-skills programme, having attendance extracted by someone blind to condition, and collecting independent fidelity and expectancy ratings. If those cannot be added, the claim should be narrowed to 'a volunteer-delivered programme was associated with lower stress' rather than 'mindfulness reduces stress'.
3. The draft claims attendance improved. Identify the inferential error and rewrite the claim in a way that matches the evidence.
Read the worked answer
The error is treating a non-significant result as if it supported the intervention. A p value of .21 does not show that attendance improved; it shows that the observed 3.1 percentage-point difference is compatible with no effect and with a range of positive and negative effects. The direction of the difference is not evidence of an effect, especially when the study is underpowered at the cluster level and attendance is a secondary outcome. The draft also inflates a single trial into a general claim about the programme. A defensible rewrite would be: 'Attendance was 3.1 percentage points higher in intervention classes, but this difference was not statistically reliable (p = .21) and the study was not powered to detect small attendance effects; the confidence interval included both no effect and meaningful benefit.' The team should report the effect estimate with its confidence interval, avoid causal language for the secondary outcome, and avoid combining a stress finding and an attendance finding into a single 'reduces stress and improves attendance' conclusion.