The Laptop Ban
A district official announces: 'Our data show that Grade 9 learners who use laptops in class score 11 points lower on the end-of-year maths test than learners who do not. We are banning laptops.' A journalist asks a researcher to comment. The dataset covers 800 Grade 9 learners in 12 schools. Laptop use was self-reported. The 240 laptop users were much more likely to attend the three schools that run a part-time work programme (62% versus 18% among non-users), and more likely to report over 10 hours of paid work per week. Test administration, marking and the sampling frame are sound. The claim rests on a single year of data.
1. Identify the main threat to the official's conclusion and explain why it matters for the ban.
Read the worked answer
The main threat is confounding: a third factor that plausibly causes both laptop use and lower marks.
- The laptop users are concentrated in the three schools running a part-time work programme (62% vs 18%), and they report more paid work hours. Paid work plausibly reduces study time and hence test scores, independently of laptops.
- School-level differences (the three programme schools may differ in staffing, timetabling or intake) are also bundled into the laptop group.
Why it matters for the ban: if the 11-point gap is largely produced by work hours and school context, banning laptops removes a study tool while leaving the actual cause untouched - a cost with no benefit. The official has established an association between two variables, and has then treated it as an effect of one on the other.
Secondary threats worth one line each: reverse causation (struggling learners may be more likely to be issued laptops for support) and self-report of laptop use.
2. The official replies: 'The test was marked reliably and the sample is large, so the finding is real.' How should the researcher respond?
Read the worked answer
Reliability of measurement is not validity of inference - they are separate claims.
- Accurate marking and a large sample tell us the 11-point difference was measured precisely. They say nothing about what produced it. A difference can be measured flawlessly and still be entirely due to a confounder.
- Sample size only narrows the uncertainty around the association. It does not convert an observational association into a causal one; with n = 800, a biased comparison is a precisely estimated biased comparison.
- Self-reported laptop use means even the exposure measure is imperfect, so the reliability argument is weaker than it sounds: the marks are reliable, the classification into 'user' and 'non-user' is not independently verified.
A fair summary for the journalist: the study is competently executed as a description, and the causal conclusion is not supported by it. Both statements can be true at once, and saying only the second is unfair to the official's work.
3. Propose one analysis or design change that would make the causal claim defensible, and state its main limitation.
Read the worked answer
Proposal: compare laptop users with non-users within the same school (school fixed effects), and additionally control for reported paid-work hours.
- The part-time work programme is a school-level feature, so comparing within schools removes that entire confound by construction. Adding hours worked addresses the individual-level part.
- If the gap shrinks towards zero once comparisons are within-school, the confound explanation is supported and the ban loses its justification. If an 11-point gap survives, the claim is much stronger - though still not proven.
Main limitation: within-school comparison discards the between-school variation, so the effective sample is smaller and the estimate less precise. It also cannot remove unmeasured individual differences - motivation, prior attainment, home resources - which may drive both laptop use and marks. Selection into laptop use is still self-chosen.
The design that would settle it: a randomised pilot in which some classes are issued laptops and comparable classes are not, with prior attainment recorded. Even then, one year of data cannot separate a genuine effect from a novelty or disruption effect in the first year of use - a second year, or a delayed-start comparison group, would be needed.