Evaluating a proposed trial of a Study Smart programme
A large Auckland secondary school has 180 Year 11 students, of whom 60 volunteered for a new six-week 'Study Smart' programme of after-school workshops run by an outside provider. The head of mathematics proposes this design: the first 30 volunteers in order of sign-up take the programme in period 5 on Tuesdays; the other 30 keep to their normal timetable and act as the comparison group. Success will be judged on the end-of-term algebra test, marked by the students' own teachers. Staff note that period 5 is when the top-band mathematics class meets, and that the provider's contract is up for renewal in December. Nothing else is planned: no pre-test, no extra sessions for the comparison group, and no change to how tests are marked.
1. Identify the main weaknesses in the proposed design and name the specific bias or problem each one creates.
Read the worked answer
- No random allocation. The first 30 to sign up are, almost by definition, the most motivated and organised. Any later difference could be motivation, not the programme - selection bias.
- Confounding with class band and timetable. Period 5 is the top-band class period. If programme students are drawn from or timetabled into that band, or if the programme displaces normal top-band teaching, prior achievement and timetable are tangled with the treatment. The comparison would not isolate the programme.
- No baseline measure. Without a pre-test or prior results, there is no way to show the groups started equal, and no way to analyse change rather than raw post-test score.
- Unblinded marking by the students' own teachers. Teachers know who attended and have an interest in the programme (and the provider's contract) succeeding - expectancy and measurement bias.
- No attention control. The programme group gets structured adult attention, a routine and a reason to study. That alone can raise scores, regardless of content.
- Volunteer sample. Conclusions apply to volunteers, not to Year 11 generally - and certainly not to the 120 students who did not volunteer.
- Possible Hawthorne effect. Students know they are in a trial and may work harder because of it.
2. Redesign the study so that a difference in algebra scores could reasonably be attributed to the programme. Justify each choice.
Read the worked answer
Identify the pool: all 60 volunteers are the eligible group; the 120 non-volunteers are not part of the trial.
Measure first: collect a pre-test (or the previous year's algebra results) for all 60 before any allocation. This gives a baseline and allows analysis of change, or use of the pre-test as a covariate, which substantially increases precision.
Stratify, then randomise: split the 60 into three prior-achievement bands of 20 and randomly allocate 10 in each band to the programme and 10 to comparison. Stratifying guarantees balance on the variable most strongly related to the outcome; randomisation within bands handles everything else, including confounders nobody measured (attitude, home support, tutor access).
Give the comparison an equal-attention condition: a supervised study session of the same length, at the same time of day, with the same access to rooms - otherwise the programme's advantage may just be extra time on task.
Control measurement: identical test, sat at the same time, scripts de-identified before marking, marked to a common schedule by teachers who do not know group membership, with a sample double-marked for moderation. Pre-specify the primary outcome (algebra score) and the analysis.
Keep allocation concealed until after the pre-test so that allocation cannot be influenced by baseline results.
A wait-list design - the comparison group receives the programme after the post-test - is fair and may improve recruitment, but it means no long-term comparison group.
3. Suppose the programme group ends up 8 marks higher on average. What can and cannot be concluded, and how should this be reported?
Read the worked answer
What can be concluded: because allocation was random, the two groups should be similar on average in prior achievement, motivation, home background and unknown factors. So a difference of 8 marks is more plausibly caused by the programme than in the original design. That is the key gain from randomisation: it addresses both measured and unmeasured confounders without needing to list them.
What still cannot be concluded:
- Chance. With 30 per group, an 8-mark difference may still be within the noise. A confidence interval for the difference (or a test) is needed; if the interval includes zero, the result is inconclusive.
- Generalisability. This is one school, one term, volunteers only. The programme may work differently with reluctant students or in a school with a different timetable.
- Longevity and transfer. The outcome is one algebra test at the end of the term; it says nothing about retention or performance in other topics or subjects.
- Implementation dependence. The result is tied to this provider, these workshops and this group size.
Reporting: state the estimated difference with a confidence interval, state the allocation method and the pre-test adjustment, acknowledge who was in the trial, and avoid causal language broader than the design allows - e.g. 'among Year 11 volunteers at this school, the programme raised end-of-term algebra scores by an estimated 8 marks (95% CI 2 to 14)'. Report the pre-test comparison too, so the reader can see the groups started comparable.