The Reading Pilot the District Wants to Scale
A district pilots a new Tier 1 reading program in four of its twelve elementary schools. The four schools volunteered; the other eight continue the current program. After one year, the treatment schools gain 9 NCE points on the state reading test, while the comparison schools gain 3. Treatment schools received two days of professional development, a half-time interventionist each, and new decodable texts. Eighteen percent of students in treatment schools left the district or moved during the year, compared with 7% in comparison schools; the district reports scores only for students who were continuously enrolled. The pilot coordinator says, 'This proves the program works, and we should buy it for all twelve schools.' The school board asks for an independent review before voting.
1. Identify the threats to internal validity that could produce the 9-versus-3 difference and explain each.
Read the worked answer
Selection is the largest threat: the four schools volunteered, so they may differ in motivation, leadership, or student composition. Resource confounding is equally serious: the program came bundled with two days of professional development, a half-time interventionist, and new texts, so the 9-point gain cannot be attributed to the program alone. Differential attrition matters: 18% versus 7%, with only continuously enrolled students counted, can inflate treatment scores if lower-performing students left. History and testing could contribute if the state test changed or if teachers practiced the format. Regression to the mean is possible if the four schools were chosen because they were low. Teacher effects are confounded because teachers were not randomly assigned, and a single enthusiastic principal can move a building's scores.
2. What additional analyses of existing data would you request before the board votes?
Read the worked answer
Request baseline equivalence on prior achievement, demographics, and mobility. Ask for attrition analyses comparing leavers and stayers by achievement in each condition, and for results under conservative assumptions about missing students. Request effect sizes with confidence intervals, not just NCE gains, plus implementation fidelity and dosage data. Check whether the interventionist was used for the program or for other purposes, and whether the comparison schools changed anything. Also check the reliability of the outcome measure and whether the same test form was used. If baseline differences explain the gap, the board should not scale the program on this evidence; if the gap survives those checks, it is still a package effect, not a program-only effect.
3. If a proper evaluation is run next year, what design and reporting standards would you recommend?
Read the worked answer
Randomize at the school level if politically feasible, with enough schools to detect a plausible effect; otherwise use matched comparison schools with a pre-specified matching plan. Pre-register the primary outcome, collect baseline and follow-up, and analyze by intent to treat. Separate the program from the supports, or test them as a package and say so explicitly. Report attrition, fidelity, and effect sizes with confidence intervals, and have testing administered independently of the program staff. Include a multi-year follow-up if the outcome is supposed to persist. Then the board can decide with a defensible estimate rather than a volunteer comparison and a bundle of resources.