Did the Training Cause the Jobs?
An NGO evaluates a 12-month vocational training programme for unemployed youth in three towns. It surveyed 500 participants at baseline and 360 at endline; 40 of the endline respondents had moved to another region. The endline shows 58% of respondents employed, compared with 22% at baseline. There was no comparison group, but the NGO notes that the national youth unemployment rate fell from 31% to 27% over the same period. The survey asked respondents whether they had 'worked for pay in the last seven days'. Town C had a flood that closed the training centre for six weeks, and 70% of its participants dropped out. The report concludes that the training caused a 36 percentage-point increase in employment and should be scaled nationally.
1. Assess the report's conclusion that the training caused a 36 percentage-point increase in employment.
Read the worked answer
The 36-point figure is an unadjusted before-after difference, not a causal effect. The same respondents were not necessarily surveyed: 500 began and 360 were at endline, so the 58% may reflect who remained. Those who found jobs may have been more likely to stay, while migrants left. The national unemployment rate fell from 31% to 27%, so some improvement would have happened without training. There is no comparison group of similar youth who did not train.
The employment question is narrow and self-reported; it counts any paid work in seven days, including casual or informal work, and does not measure job quality or income. Town C's flood and 70% dropout mean the programme was not delivered as designed there.
The report should say that employment among surveyed participants rose by 36 points, not that the training caused it. A credible causal claim needs a counterfactual, consistent follow-up and analysis of attrition.
2. Identify the main threats to internal and external validity and explain how each could affect the estimate.
Read the worked answer
Internal validity threats include selection: participants may be more motivated than non-participants. Attrition is 28% overall and likely higher in Town C; if dropouts differ, the endline sample is biased. History is a threat because national unemployment fell and the flood affected Town C. Maturation may also matter as youth find work over time.
Self-report and testing can bias results: repeated questions may encourage respondents to report work, and 'worked for pay in the last seven days' may capture informal casual work inconsistently. Instrumentation may change if survey staff or questionnaires differ.
External validity is limited because three towns and a 12-month programme may not represent other regions; scaling nationally assumes similar labour markets. The estimate could be overstated if the most employable stayed, or understated if migrants found jobs elsewhere and were lost. Town C should be analysed separately or the results weighted, and the report should present confidence intervals and attrition by town.
3. Propose a revised evaluation design and analysis plan that the NGO could use in a second phase.
Read the worked answer
A second phase should use a comparison group. If randomization is ethical and feasible, randomize eligible youth to training now or to a delayed cohort, and compare employment using intention-to-treat. If not, use propensity-score matching or a difference-in-differences design with comparable non-participants in the same towns.
Define employment and earnings precisely, including casual and informal work, and verify a subsample with employers or receipts. Track all baseline respondents, including migrants, by phone and community contacts; report attrition and use inverse-probability weights. Stratify by town and analyse Town C separately because of the flood. Collect baseline data on motivation, education and household income to control for selection.
Power the study for a plausible effect, not the pilot's inflated estimate. Pre-register the analysis and report both intended and actual programme delivery. This design would cost more but would let the NGO make a defensible scaling decision.