School Phone Ban and Causal Inference
Westbrook Secondary School has 640 students in grades 9–12. At the start of the year it banned smartphones during lessons and breaks for grades 9–11 (480 students); grade 12 (160 students) was exempt because of exam preparation. The school also launched a free after-school tutoring programme for all grades in the same term. At year end, average test scores in the banned grades rose from 62% to 68%; grade 12 rose from 64% to 69%. Absences in the banned grades fell from 9% to 7%, while grade 12 absences fell from 8% to 7.4%. The principal announced: 'The phone ban raised attainment by six percentage points and cut absenteeism by two points; we should extend it to grade 12.' A parent replies that the comparison proves nothing because grade 12 also improved.
1. Using the grade 12 data, evaluate the principal's causal claim.
Read the worked answer
Compare the change in the banned grades with the change in the exempt grade. Banned grades: 62 → 68, +6 points. Grade 12: 64 → 69, +5 points. The difference-in-differences estimate is +1 point, not +6. For absences, banned grades fell 2.0 points; grade 12 fell 0.6 points; the difference is 1.4 points. The principal's claim attributes the entire 6-point rise to the ban, but grade 12 improved almost as much without a ban. That pattern suggests a common cause - most plausibly the tutoring programme launched in the same term, or a change in tests, teaching, or cohort. The ban may have had a small effect, but the data do not support a 6-point causal claim. A control group that improved by 5 points is exactly what undermines the naive before-after comparison.
2. Identify two alternative explanations for the improvement and state what evidence would distinguish them from the ban.
Read the worked answer
Alternative explanations include: (a) the after-school tutoring programme, which applied to all grades and could raise scores and attendance; (b) a change in test difficulty or marking, which would move both groups; (c) teacher changes, cohort differences, or regression to the mean if the prior year was unusually low; and (d) general school-wide attention to attendance. To distinguish them, the school could compare with previous years' trends, check whether gains were larger among students who attended tutoring, and randomly assign the ban to some classes while keeping others exempt. A difference-in-differences with a genuine control and pre-treatment trends would be stronger than a single before-after comparison.
3. The principal says extending the ban to grade 12 is justified because 'even one percentage point is worth it.' What is the flaw, and what would you need to know?
Read the worked answer
The flaw is treating a 1-point difference-in-differences as a reliable causal effect and then claiming even one point is worth it. A 1-point difference is well within sampling noise at these group sizes; the school has 480 and 160 students, and test scores vary. Before extending the ban, the school would need confidence intervals, evidence that the 1-point estimate is stable across subjects and years, the cost of enforcing the ban in grade 12, and whether phone use supports exam preparation. It would also need to know whether the ban's benefit comes from reduced distraction or from stricter discipline in general. 'Worth it' is a normative claim requiring a cost-benefit comparison, not just a positive point estimate.