Evaluating a tutoring programme without random assignment (general education: research methodology)
A master's student in education wants to show that an 8-week after-school mathematics tutoring programme raised scores in rural schools. Twelve schools were invited; three volunteered. In each of the three, the programme was offered to all 9th-graders who chose to attend: 118 students enrolled, while 74 did not. The student plans to compare the enrolled and non-enrolled students' end-of-term test scores and, if the enrolled group is higher, to claim the programme caused the gain. The department has no budget for random assignment, but the student has access to each student's prior-term mathematics mark, attendance, and free-school-meal status. The defence is in 3 weeks, and the committee has asked for a design that can support a causal claim 'as far as possible'.
1. Identify the main threats to internal validity in the planned comparison and explain, with reference to the numbers, how each could produce a false positive.
Read the worked answer
Selection: the 3 volunteer schools and the 118 self-selected students may already be more motivated or better supported. If the enrolled group's prior-term mean is higher, the comparison confounds the programme with prior achievement.
Self-selection: families who choose after-school tutoring may also supervise homework; the 74 non-enrolled are not a counterfactual for the 118.
Maturation: over 8 weeks all students receive normal mathematics instruction; end-of-term gains may reflect the curriculum, not the programme.
Testing/practice: the end-of-term test may resemble tutoring materials, giving the enrolled group a format advantage.
Regression to the mean: if students enrolled because of a low prior mark, their scores would rise somewhat without the programme.
Differential attrition: if weaker enrolled students drop out, the remaining 118 look better.
Instrumentation: a new end-of-term test or a change in marking can create differences.
Teacher/classroom effect: the tutoring may be delivered by a particularly effective teacher. With 118 versus 74, a 0.3 SD difference could look statistically significant even if it is entirely selection.
2. Propose a feasible quasi-experimental design and the analysis you would defend before the committee. State the key limitation in one sentence.
Read the worked answer
Design: a matched comparison using the 74 non-enrolled students and, if possible, non-participating students in the same schools. Match on prior-term mathematics mark (within 5 points), attendance band, and free-school-meal status. Use exact matching where cells are large enough, otherwise propensity scores. Check balance: standardized mean differences below 0.10 for all covariates, and inspect common support.
Analysis: ANCOVA of end-of-term score with prior mark, attendance, and free-school-meal status as covariates, plus school fixed effects if there are enough students per school. Report effect size and 95% CI. Secondary check: within the enrolled group, test whether hours attended (dosage) predict gain while controlling for prior mark - a dose-response pattern is more credible than a simple group difference. Sensitivity: calculate an E-value for unmeasured confounding.
Limitation: matching cannot rule out unmeasured motivation or parental support, so the strongest defensible claim is 'associated with higher scores after adjustment', not 'caused'.