From 7 points to a defensible claim: rewriting a causal memo
A ministry memo circulated for budget approval states: 'The school feeding programme raised learning by 7 points. Schools in the programme averaged 62 on the national test, against 55 in non-programme schools, so the programme should be scaled nationally.' The background is as follows. One hundred and eighty schools applied; 120 were funded, ranked by a poverty index with a published cutoff of 0.62 (higher means poorer). Participation was voluntary. Thirty of the funded schools are missing post-test scores because they did not administer the test, and these are concentrated in remote districts. The national test has a standard deviation of 18 points. The budget committee has asked for a two-page note on whether the claim is defensible and what evidence would settle the question, written for non-specialist members of parliament.
1. Identify the main threats to the memo's inference and state the likely direction of each bias.
Read the worked answer
Threat 1 - selection into participation. Schools applied voluntarily, so applicants differ from non-applicants in motivation, management quality and parental engagement, all of which raise test scores independently of feeding. Comparing participants with the 55-point group (which includes schools that never applied) attributes that gap to the programme. Direction: upward bias.
Threat 2 - selection on the funding rule. Within applicants, funding was ranked by poverty, so funded schools are poorer than unfunded applicants. Poorer schools typically score lower for reasons unrelated to feeding. Direction: downward bias, which partly offsets threat 1 and makes the net sign of the 7-point gap uninterpretable.
Threat 3 - attrition. Thirty funded schools, concentrated in remote districts, have no post-test. If remote schools are poorer and weaker, dropping them removes low scores from the treated mean. Direction: upward bias, plausibly large: 25% of the treated group is missing.
Threat 4 - a conflated comparison group. The memo compares programme schools with 'non-programme schools' without saying whether these are unfunded applicants or all other schools. The two comparisons answer different questions and would give different numbers.
Overall. Three biases, two upward and one downward, cannot be signed in aggregate. The 7 points is a descriptive difference, not an effect, and the memo's causal verb is unsupported. The 18-point standard deviation also means 7 points is about 0.39 SD - plausible in magnitude, which is exactly why the design matters.
2. Explain the regression-discontinuity design available here, the assumption it needs, and the checks that would make it credible to a sceptical committee.
Read the worked answer
The design. Funding is determined by the poverty index with a sharp cutoff at 0.62: schools above it were funded, schools below it were not (among applicants). Compare schools just above and just below 0.62. These schools are similar in poverty and in unobserved characteristics, differing only in whether they cleared the threshold by a small margin, so the jump in test scores at the cutoff identifies the effect for schools near the threshold.
Key assumption. Potential outcomes must be continuous in the poverty index at 0.62: absent the programme, scores would not jump at exactly 0.62. The design fails if schools can manipulate the index or if another programme uses the same cutoff.
Credibility checks. (1) McCrary density test: is there bunching of schools just above 0.62, suggesting manipulation of reported poverty data? (2) Covariate balance: do baseline characteristics (enrolment, teacher ratios, pre-test scores) move smoothly through the cutoff? (3) Placebo cutoffs: no jump at 0.55 or 0.70. (4) Bandwidth sensitivity: report the estimate at several bandwidths and with a bias-corrected local linear estimator. (5) Attrition check: is missingness smooth at the cutoff? If remote schools above the cutoff drop out, the design inherits the attrition bias.
Interpretation limit. The estimate applies to schools near the threshold, not to all 180 applicants, and certainly not to all schools nationally. Scaling nationally is an external-validity claim that the RD cannot support on its own.
3. Rewrite the memo's causal sentence as a calibrated academic-English summary of two to three sentences, suitable for non-specialist readers, and state what additional data would settle the question.
Read the worked answer
Rewrite. 'Among the 150 funded and unfunded schools with complete test data, schools in the programme scored 7 points higher on average than those outside it - about 0.4 of a standard deviation. Because schools applied voluntarily, funding was targeted at poorer schools, and a quarter of funded schools are missing results, this difference cannot be interpreted as the effect of the programme. A comparison of schools just above and just below the funding threshold of 0.62 would provide a more credible estimate, but only for schools close to that threshold.'
Why this version is better. It states the comparison and the magnitude, names the three threats in plain language, and replaces the causal verb 'raised' with a description of the difference. It avoids jargon ('endogeneity', 'selection on observables') while keeping the technical content accurate, and it signals the limits of the proposed design. Word choice matters: 'associated with' is defensible; 'raised' and 'should be scaled' are not.
Data that would settle it. Baseline test scores for all applicants, so pre-programme achievement can be differenced out; the full poverty index for every applicant (not just the funded), to allow the RD and the density test; post-test scores and a reason for non-administration for the 30 missing schools, to bound attrition; and the funding rule documentation, to confirm the cutoff is sharp and no other programme shares it. With these, the committee could be offered a local RD estimate with a randomisation or bias-corrected confidence interval, plus an explicit statement of the population to which it applies.