Scaling a pilot on weak evidence
A youth mental health NGO runs an eight-week peer support pilot in three schools. Sixty students enrol; 47 complete the pre- and post-survey; scores on a wellbeing scale improve by an average of 6 points. The same staff deliver the program and collect the surveys. There is no comparison group. Students self-select into the program, and teachers nominate some participants. Attendance records show that the most distressed students missed more sessions. The board wants to claim the program is 'proven' and roll it out to 200 schools. A funder asks for an evaluation plan before committing.
1. Evaluate the board's claim that the program is 'proven'. Identify the main threats to validity.
Read the worked answer
The claim is too strong. A pre-post design with no comparison group cannot separate program effects from maturation, seasonal change, regression to the mean or other school events. Self-selection and teacher nomination create selection bias: motivated or less distressed students may improve. The same staff deliver and collect data, risking social desirability and assessor bias. Thirteen of 60 did not complete; if the most distressed missed sessions, the 6-point average may overstate benefit and hide harm. A wellbeing scale is self-report and may be unreliable. Attendance data suggest dose matters.
The result is promising but not proof. The board can say the program was 'associated with improvement in completers', not 'proven' or ready for statewide rollout. A credible comparison and independent data are needed.
2. Design a more credible evaluation the NGO could run before scaling, including ethical safeguards.
Read the worked answer
Use a staggered or waitlist design: allocate schools or students to immediate or delayed program by lottery or matched comparison. Use independent data collectors, a validated measure plus attendance/dose, baseline, post and three-month follow-up, and track non-completers. Include qualitative interviews about mechanisms and harms. Pre-register primary outcomes and the analysis plan.
Ethics: obtain informed consent and student assent; protect confidentiality; provide a referral pathway for distressed students; ensure cultural safety; do not withhold usual care; debrief participants.
Feasibility: pilot in 6–10 schools with enough power to detect a realistic effect. If randomisation is unacceptable, use a matched comparison and control for baseline. Report limitations. Scale only after independent review, and monitor equity so the most distressed are not excluded.