Evaluating a Reading-Tablet Pilot Before a PKR 1.9 Billion Scale-Up
A provincial education department reports that its reading-tablet pilot raised Class 4 Urdu scores by 12 marks in the 40 schools that received tablets, against 4 marks in 40 other schools over the same eight months. It proposes to scale the programme to 2,000 schools at PKR 1.9 billion.
An analyst reads the file and notes four things. First, the 40 pilot schools were selected precisely because they already had functional computer rooms and a head teacher rated 'active' by the district office; the comparison schools were simply the remaining schools in the same districts. Second, tablet delivery slipped three months, so actual exposure was five months, not eight. Third, six of the 40 pilot schools never used the tablets at all because no one trained the teachers - yet their scores rose by 11 marks. Fourth, the tests were marked by the project's own monitors, who knew which schools were in the pilot. No baseline was published.
1. What does the 11-mark rise in the six unused-tablet schools imply about the reported 12-mark effect? Reason through the alternative explanations.
Read the worked answer
The six unused-tablet schools are the most important finding in the file.
If tablets caused the gain, schools that never used them should show little or no gain. They show 11 marks against 12 for the pilot as a whole. The tablet's attributable contribution is therefore at most about one mark (12 − 11), and a one-mark difference across a 40-school sample with unpublished baseline is well inside ordinary noise. The headline '12 versus 4' survives only if the reader ignores the internal comparison the file itself supplies.
Alternative explanations, each supported by the file:
- Selection. Pilot schools were chosen for functional computer rooms and 'active' head teachers - characteristics independently associated with faster improvement. The comparison schools were simply the leftovers in the same districts, so the contrast measures school quality as much as tablets.
- Tester bias. Monitors who knew which schools were in the pilot marked the tests. Expectancy effects on a subjective scoring task of this kind are not a small correction.
- Attention and novelty. Being supplied, visited and monitored raises performance on its own, whether or not the device is used.
- Missing baseline. Without one, the 4-mark comparison gain may understate normal progress in schools that were already improving, or overstate it if the groups differed to begin with.
- Exposure mismatch. With a three-month delivery slip, the claimed effect covers five months but the comparison covers eight - the two arms were not measuring the same period of change.
Conclusion: the file does not show that tablets raise learning. It shows that well-resourced, closely watched schools improved, which is a different claim and a much cheaper one to act on.
2. What single additional comparison would most cheaply test the claim, and what would it cost to mount?
Read the worked answer
Create a control inside the eligible population.
The 40 pilot schools were eligible only because they had functional computer rooms and an 'active' head teacher. So compare them with schools that share those two characteristics and did not get tablets - or simply match comparison schools to pilot schools on those two variables using data the department already holds. This removes the selection explanation in one stroke, at almost no cost.
If the scale-up is going ahead regardless, randomise the order of delivery. With 2,000 schools, a phased rollout randomised at school level gives a credible counterfactual for free - later cohorts act as controls for earlier ones - and the department is spending this money anyway. It is the single cheapest upgrade available.
Two supporting fixes:
- Blind the scoring. Tests marked by teachers from another district, with school identity removed.
- Use the natural third arm that already exists: the six untrained, non-using pilot schools. Trained-use, untrained-no-use and matched control gives an informative three-way contrast from a single testing round.
Cost: one additional testing round, independent markers, and analysis - a small fraction of PKR 1.9 billion, and far less than a single year of tablets. The point is not to demand a perfect trial; it is to insist that the department can say in advance what result would prove it wrong.
3. Should the department scale to 2,000 schools now? Give a decision with conditions, including what the unit economics require.
Read the worked answer
No - not on this evidence, at this price, in one step.
Why the evidence fails. The one internal comparison available (unused schools gaining 11 marks against 12) points to the schools and the attention, not to the devices. No baseline is published, the markers were not blind, exposure was five months against a comparison of eight, and delivery slipped - so even the direction of the reported differential is uncertain.
Why the unit economics need testing before, not after. PKR 1.9 billion ÷ 2,000 schools ≈ PKR 950,000 per school. Before committing that, the department should publish what the pilot actually cost per school - tablets, teacher training, connectivity, replacement and maintenance - and compare it with alternatives that already have evidence behind them, such as additional teachers, libraries or remedial coaching. A per-school figure near a million rupees is an unusually large bet to place on a one-mark difference.
Decision: approve a phased, randomised rollout with a conditional release.
- Start with about 200 schools, assigned at random to early and later cohorts, with baseline and endline testing, blind marking, and an analysis plan registered before the results are seen.
- Release the balance only if the effect survives the matched comparison.
- Fix the training gap first: six of forty schools untrained is a design failure, not a tablet failure, and would be repeated at scale if not addressed.
- Integrate the devices into the existing curriculum, and publish results whether positive or negative.
A programme of this size should be judged by whether it can state, honestly, what evidence would stop it. On the present file, nothing could.