The Donor's 12-Point Claim
A district education office wants to know whether a 10-week after-school reading programme works. It is offered only in School A because a donor funded 60 places there; School B, 3 km away, has 62 comparable students and no programme. Both schools use the same 40-item reading test in the first week and again in week 12. School A's mean score rises from 21 to 34; School B's mean rises from 22 to 25. Attendance in the programme averages 78%, and the School A head teacher has publicly promised parents that scores will improve. Teachers in School A volunteered to run the sessions and received a small stipend. Parents in School A were told the programme was selective. The donor wants a one-page claim that the programme 'adds 12 points' and plans to expand it to 20 schools next year. The district has no additional budget for random assignment, and mid-year transfers mean the original lists are no longer identical.
1. Identify the design and evaluate whether the claim that the programme 'adds 12 points' is warranted.
Read the worked answer
This is a non-equivalent control group, pretest–post-test quasi-experiment. The independent variable is programme participation; the dependent variable is the 40-item reading score. The claim is not warranted. The observed difference-in-differences is (34−21) − (25−22) = 13 − 3 = 10 points, not 12. The 12-point figure appears to compare School A's post-test (34) with School B's post-test (25), which ignores the baseline difference and is a post-test-only comparison. Even the 10-point estimate is confounded: School A's teachers volunteered, parents were told the programme was selective, and the head teacher made a public promise. Motivation, teacher commitment and parental involvement may all differ. The design supports a hypothesis, not proof of effect.
2. Name three threats to validity in this evaluation and give one feasible fix for each.
Read the worked answer
First, selection–maturation and selection–history: School A may have had a simultaneous remedial reading drive or a more motivated cohort. Feasible fix: add a third comparable school or a waiting-list control; at minimum, match students on prior scores and background and report the comparison. Second, testing and instrumentation: the same 40-item test may be remembered, or teachers may teach to it. Fix: use a parallel form and blind scorers to condition. Third, attrition: mid-year transfers may differ by school. Fix: report attrition by group and conduct an intention-to-treat analysis, tracking all originally listed students. The head teacher's public promise also creates expectancy effects; reduce these by not telling teachers which scores are being compared and by using independent assessors.
3. What additional data should the district collect before expanding to 20 schools, and how should the press release be worded?
Read the worked answer
Collect baseline equivalence on prior reading scores, attendance, home language and socioeconomic status; dosage, since 78% attendance means 22% of sessions were missed; implementation fidelity, meaning whether volunteers followed the programme; and evidence that the test samples the taught skills. Before expansion, run a small randomised or stepped-wedge trial across 6–8 schools with a pre-registered primary outcome, parallel test forms and blinded scoring. Calculate cost per additional point gained. The press release should say 'associated with' or 'promising', not 'proven', and should report the baseline-adjusted estimate with a confidence interval. A responsible wording would be: 'In one school, scores rose more than in a comparison school; a larger trial is needed to test whether the programme caused the gain.'