The Regression That Proved the Policy Worked
A government evaluation of a conditional cash transfer reports that beneficiaries' school enrolment rose 11 percentage points relative to non-beneficiaries, significant at the 1 percent level, using a cross-sectional regression on 4,200 households with controls for income, parental education, and district. The programme was rolled out in phases: the poorest 40 percent of villages were covered first, and the remaining villages were covered two years later. The evaluation used the most recent survey, after all villages were covered, and defined 'beneficiary' as any household that had ever received a transfer. A junior economist notes that the 11-point estimate is larger than the 6-point effect found in the programme's own baseline midline survey. The ministry wants to cite the 11-point figure in its budget submission.
1. Identify the specific threats to the validity of the 11-point estimate and explain why the phased rollout matters.
Read the worked answer
Threat 1 - selection on unobservables. The programme covered the poorest 40 percent of villages first. 'Beneficiary' is therefore not randomly assigned: it is correlated with village poverty, which is correlated with school quality, distance to school, and household aspiration. Controls for income, parental education, and district cannot absorb village-level differences that are unmeasured. The comparison group is systematically different in ways the regression cannot fix.
Threat 2 - the definition of 'beneficiary' is contaminated. Defining treatment as 'ever received a transfer' after all villages were covered means the comparison group is partly treated or affected by spillovers. If the later-covered villages were also eligible, the 'non-beneficiary' group is a mix of ineligible households and households in treated villages. This biases the estimate in an unknown direction and makes the 11-point figure not comparable to the 6-point midline estimate, which was measured at a time when the comparison group was genuinely untreated.
Why the phased rollout matters. A phased rollout is a natural experiment. The correct design compares early-covered villages with late-covered villages during the period when the late villages were not yet covered - a difference-in-differences or, better, an instrumental-variables or regression-discontinuity design using the phase assignment. That design uses the rollout as the source of variation and avoids comparing beneficiaries with people who were themselves treated or eligible.
The cross-sectional regression throws away exactly the variation the rollout provides. It is the wrong design for the data, and the 11-point estimate is best read as a selection-contaminated upper bound.
2. The ministry wants the 11-point figure. Draft the technical note you would send, stating what can and cannot be claimed, and what analysis should be run before the budget submission.
Read the worked answer
Technical note: interpretation of the enrolment effect estimate
What cannot be claimed. The 11-percentage-point estimate cannot be presented as the causal effect of the transfer. It compares households that ever received a transfer with a comparison group that, after full rollout, includes treated and eligible households. The estimate is also larger than the programme's own midline estimate of 6 points, and that gap is a warning sign, not a bonus: the most likely explanation is selection, because the first-covered villages were the poorest and the regression controls cannot absorb village-level differences.
What can be claimed. The data show a strong positive association between transfer receipt and enrolment. That is consistent with a real effect but does not establish its size.
Analysis required before submission.
- Restrict to the rollout window. Use the phase assignment (early vs late villages) and compare enrolment in the period before the late villages were covered. This is a difference-in-differences design and is the minimum acceptable specification.
- Use the phase as an instrument for receipt, to address the selection problem directly, and report the first-stage strength.
- Test for pre-trends. If early and late villages had different enrolment trajectories before the programme, the parallel-trends assumption fails and the DiD estimate is not credible.
- Reconcile with the midline. If the DiD estimate lands near 6 points, the 11-point figure is confirmed as selection-biased and should be withdrawn. If it lands near 11, the midline estimate needs re-examination.
- Report standard errors clustered at the village level, since treatment is assigned at village level; household-level clustering will overstate precision.
Recommendation. Do not cite 11 points in the budget submission. Cite the midline 6-point figure as the current best estimate, and commit to the DiD re-analysis within the quarter. Citing a number the ministry's own baseline contradicts invites exactly the scrutiny that will discredit the programme.