The Vanishing Effect: When a Pilot Study Refuses to Scale
Priya is investigating whether a 10-minute nature walk improves sustained attention in Year 11 students. In her pilot with 18 volunteers, mean reaction time on a 5-minute vigilance task improved by 42 ms after the walk versus a rest condition, and she is excited. For the full study she recruits 120 students but must run them in four groups of 30 across four lunchtimes because the school hall is only free then. Group 1 runs on a Monday after a public-holiday weekend, Group 2 on a stormy Thursday, Group 3 the day after a whole-school athletics carnival, and Group 4 during end-of-term assessments. Her teacher notes that the pilot group was self-selected from her psychology class. Priya's draft report claims a 'consistent attention benefit of nature exposure' and she has already built her oral presentation around that headline.
1. Identify the two most serious threats to the validity of Priya's full study and explain how each could plausibly produce her result.
Read the worked answer
Threat 1: Confounding by testing occasion. Condition (walk vs rest) is not separated from the day on which each group is tested. Group 1 follows a holiday weekend, Group 3 follows an athletics carnival, Group 4 sits during assessments - each of these plausibly shifts baseline arousal, fatigue, or motivation, which directly affects vigilance reaction time. If the walk condition happened to be scheduled on the calmer days, the 42 ms difference could be occasion, not walking.
Threat 2: Self-selection and non-equivalent groups. The pilot drew volunteers from Priya's psychology class, and the full sample is likewise volunteers. Students who volunteer for a nature-walk study may already differ in attentiveness, outdoor affinity, or motivation to please the researcher. Without random allocation to condition within each group, any difference is correlational.
A third, weaker threat is order effects if every participant always rests then walks; fatigue or practice on the vigilance task would inflate the walk condition.
2. Recommend a concrete redesign that Priya could realistically run within the four lunchtime slots, and state what she may and may not claim afterwards.
Read the worked answer
Redesign. Within each of the four groups of 30, randomly allocate 15 to walk-first and 15 to rest-first, and counterbalance the order (walk–rest vs rest–walk) so practice and fatigue are spread across conditions. Hold the vigilance task duration, time of day, and instructions constant; record and report the day-level context (post-holiday, storm, post-carnival, assessment period) as a covariate or at minimum as a limitation. Pre-register the primary outcome (mean reaction time) and the analysis before running.
What she may claim. With randomisation within groups, she can claim a causal effect conditional on the tested occasions, and she should report the day-by-day pattern rather than a pooled 'consistent benefit'. She may not generalise to all Year 11 students, nor claim consistency across contexts, because four non-equivalent days cannot support a consistency claim. The honest headline is conditional: 'under these four testing occasions, with randomised order, the walk condition showed X'.
3. Priya's oral presentation is built on the phrase 'consistent attention benefit'. Evaluate whether this framing is defensible and describe how she should restructure the presentation.
Read the worked answer
The phrase is not defensible on her evidence. 'Consistent' asserts stability across occasions, but her four occasions differ systematically and she has no within-occasion replication. The presentation should instead lead with the conditional finding, show the four group results side by side so the audience can see the variability, and explicitly name occasion and self-selection as limits. A strong structure: (1) question and why it matters, (2) design including randomisation and the four occasions, (3) results per group with variability visible, (4) interpretation with the two threats stated, (5) what a follow-up would need. This is more persuasive to an assessor than an overclaimed headline, because it demonstrates that she understands the inferential boundary of her own data.