The Tutoring Program Comparison
A student evaluates a peer tutoring program. Twenty volunteers were assigned to tutor 40 struggling students. At the end of the term, tutored students' average quiz score rose from 62 to 71, while a comparison group of 40 students who were not tutored rose from 64 to 70. The student concludes the program caused a 9-point gain. A teacher points out that students were assigned to tutoring based on teacher recommendation, not randomly, and that tutored students may have been more motivated to begin with. The student also notices that 6 tutored students missed more than half the sessions, and that the comparison group took a slightly easier quiz.
1. Evaluate the student's causal claim. What does the data support, and what does it not support?
Read the worked answer
The claim that the program "caused a 9-point gain" is not supported. The 9-point figure is a pre-post change within the tutored group, not a comparison against a valid counterfactual. The comparison group gained 6 points (64 to 70), so the raw difference in gains is about 3 points, not 9. But even that 3-point difference is fragile: assignment was based on teacher recommendation, so selection bias is likely - tutored students may have differed in motivation, prior achievement, or need. The comparison group also took a slightly easier quiz, which could inflate their gain and shrink the apparent program effect (or distort it in either direction). What the data support is descriptive: tutored students' scores rose on average, and their average gain exceeded the comparison group's by about 3 points on non-equivalent quizzes. A causal claim would require random assignment or a strong quasi-experimental design, plus equivalent outcome measures.
2. How should the 6 students who missed most sessions be handled in the analysis, and what should the student report?
Read the worked answer
There is no single correct answer, but the choice must be stated and justified. Option 1: analyze all tutored students as assigned (an intention-to-treat approach), which answers "what happened when the program was offered" and avoids the bias that comes from dropping unmotivated students. Option 2: analyze only students who attended most sessions (a per-protocol approach), which estimates the effect of actually receiving tutoring but risks selection bias, since attenders may differ from non-attenders. The student should not silently drop the 6; that would inflate the apparent effect and hide a real implementation problem (low attendance). Best practice: report both, clearly labeled, and describe attendance as part of the findings. She should also note that with 40 tutored students, 6 is 15% - a substantial share whose experience the program did not reach. Reporting both analyses is more honest than choosing the one with the larger effect.
3. The comparison group took a slightly easier quiz. Explain how this affects interpretation and what the student should do about it.
Read the worked answer
If the comparison quiz was easier, the comparison group's gain (6 points) may be inflated relative to what it would have been on the harder quiz. That would make the tutored group's 3-point advantage an underestimate of the true difference. But "slightly easier" is vague, and the direction of distortion depends on how difficulty interacts with the students' skill levels - an easier test can compress or expand differences depending on where students sit on the score range. The student cannot correct this with the data in hand. What she should do: (1) state the measurement difference explicitly as a limitation; (2) avoid presenting the 3-point gap as a precise effect size; (3) if possible, obtain item-level or equated scores, or check whether any subset of items was common to both quizzes; and (4) frame conclusions as provisional pending a design with a common outcome measure. The core lesson is that comparing groups requires comparable measures, and when they differ, the comparison weakens.