Simpson's Paradox in a Mentoring-Program Evaluation
A district compares graduation rates for students in a new mentoring program versus students not in the program. North High: 100 program students, 90 graduated; 400 non-program students, 320 graduated. South High: 400 program students, 280 graduated; 100 non-program students, 60 graduated. The district aggregates the numbers and reports that program students graduated at a lower rate than non-program students, so it proposes ending the program. A parent notes that within each school the program group has the higher rate and asks how both statements can be true.
1. Compute the graduation rates by school and the aggregate rates. Explain how the program can be higher within each school but lower overall.
Read the worked answer
North: program 90/100=90%, non-program 320/400=80%. South: program 280/400=70%, non-program 60/100=60%. The program is higher within both schools. Aggregated, program graduates are 90+280=370 of 500, or 74%; non-program graduates are 320+60=380 of 500, or 76%. The reversal occurs because program students are concentrated at South, where graduation rates are lower for everyone, while non-program students are concentrated at North. The aggregate rate is a weighted average of school rates, and the weights differ sharply between the two groups. This is Simpson's paradox.
2. What variable is confounding the comparison, and what analysis would be more appropriate before deciding whether to end the program?
Read the worked answer
School is the confounder, standing in for differences in student population, community, and school resources. A valid comparison should be stratified by school or should adjust for school-level and student-level covariates. The district should report within-school differences, with sample sizes and uncertainty, and should examine whether program participation was voluntary or assigned. If the goal is causal, a randomized or matched design is needed. Pooling the two schools without adjustment gives more weight to South's program students and reverses the within-school pattern.
3. The district wants to claim that the program causes a 10-percentage-point increase in graduation. Is that causal claim supported? Explain what evidence would strengthen it.
Read the worked answer
It is not supported. The data are observational, and participation is likely related to school, prior achievement, or student choice. The within-school 10-point differences are consistent with a benefit but do not prove causation. A stronger design would randomly assign eligible students to program or control, or use matching on prior GPA, attendance, and school. The district should also check graduation definitions and missing data. Without such controls, the safest conclusion is that program students at each school had higher observed rates, not that the program caused the increase.