Training Pilot: Interpreting an Encouragement-Design IV Against OLS
A labour ministry ran a pilot on 1,800 registered unemployed people. Half were randomly selected to receive an offer of a six-week training course; the other half were not. Take-up was imperfect: 62% of those offered enrolled, while 15% of those not offered found an equivalent course through a partner NGO. Two years later, earnings are observed for everyone. The naive comparison of attendees and non-attendees shows a gain of €1,150 per year (SE €420). The ministry's analysts instead use random assignment as an instrument for actual attendance: the reduced form of earnings on the offer is €780 (SE €260), and the first stage of attendance on the offer is 0.47 (SE 0.037). A consultant concludes that the programme 'raises earnings by roughly €1,700 a year and should be rolled out nationally', noting that the unit cost is about €6,000 per attendee. A second consultant argues the instrument is too weak to say anything.
1. Compute the instrumental-variables estimate with its standard error and 95% interval. Assess the claim that the instrument is weak, and state whether the design shows that the OLS estimate of €1,150 is biased.
Read the worked answer
Estimate. Wald/2SLS: β_IV = reduced form / first stage = 780 / 0.47 = ≈ €1,660 per attendee.
Standard error by the delta method. With both stages estimated, SE ≈ (1/0.47) × √(260² + (1,660 × 0.037)²) = 2.128 × √(67,600 + 3,777) ≈ 2.128 × 267 ≈ €568. The 95% interval is therefore roughly €546 to €2,774 - wide, and including values close to zero.
Weak instrument? No. The first-stage F is (0.47/0.037)² ≈ 161, far above the conventional threshold of about 10, so the relevance condition is comfortably satisfied and 2SLS bias toward OLS is negligible. The second consultant is wrong on this point.
Does it show OLS bias? No. The IV point estimate is about 44% larger than OLS, which is consistent with attenuation or with upward selection, but the confidence interval easily contains €1,150. Formally, the data cannot reject equality of the OLS and IV estimands. The correct statement is that the design fails to detect selection bias, not that selection bias has been ruled out.
2. State the assumptions behind the IV estimate and define the group to which it applies. Explain why the number should not be read as the effect for people who would attend training anyway.
Read the worked answer
Assumptions. (i) Relevance: the offer shifts attendance - satisfied, F ≈ 161. (ii) Independence/random assignment: the offer is independent of potential earnings - satisfied by the lottery. (iii) Exclusion: the offer affects earnings only through attendance, which requires that being offered a place but declining does not change behaviour (no discouragement or signalling effect on employers) - plausible but untestable here. (iv) Monotonicity: nobody attends only when not offered; with 62% take-up among the offered and 15% among the controls, defiers would have to be numerous to break the design, and this is a substantive assumption to defend.
Population. Because take-up is imperfect in both arms, IV identifies a local average treatment effect for compliers: the 47% of the sample who attend only when offered (62% − 15%). The always-takers are 15% of the sample, the never-takers 38%; the estimate says nothing about either group, and nothing about people who would enrol even without an offer - precisely those most motivated, and plausibly those with the highest or lowest returns. The extrapolation to 'the programme raises earnings by €1,700 nationally' silently assumes homogeneous effects, which the design cannot verify.
3. The ministry must decide on national rollout at a cost of €6,000 per attendee. Set out a decision rule and say what the current evidence supports.
Read the worked answer
Decision rule. Roll out only if the expected present value of the earnings gain to a complier exceeds the unit cost. At a real discount rate of, say, 3% and a 10-year horizon, the annuity factor is ≈ 8.5, so the break-even annual gain is 6,000 / 8.5 ≈ €706 per year. Using the lower 95% bound of €546 per year gives a present value of roughly 546 × 8.5 ≈ €4,640, below the €6,000 cost; the point estimate of €1,660 gives ≈ €14,100, comfortably above it.
The evidence is therefore inconclusive at the stated unit cost, not supportive: the sign is positive, the design is sound, but the interval spans a negative return on spending. A defensible decision rule is 'roll out if the lower confidence bound on present value exceeds unit cost'; on this rule the answer is no.
Reasonable next steps: commission a larger sample or a longer earnings panel to halve the standard error, target the rollout at subgroups with higher expected returns, or negotiate the unit cost down: at €4,000 per attendee the break-even annual gain falls to ≈ €470, below the lower bound, and rollout would pass the same rule.