Did the Workforce Program Work?
A city workforce development program served 1,800 residents over two years; 1,200 completed. Six months after exit, 68% of completers were employed, versus 61% of all city residents; average wage was $19.40 versus $18.10. The mayor wants a press release claiming a '7-point employment gain and $1.30 wage premium.' The analyst notes that participants self-selected, 300 dropped out, the comparison includes retirees and part-time workers, the follow-up survey had a 40% response rate, and two providers used different service models. Only 85 participants were under age 24, and their employment rate was 54%. The program costs $4,800 per completer. The council asks whether to expand, continue, or redesign. No random assignment was used, and intake staff say motivated applicants apply early. The council has $600,000 available for next year.
1. Assess whether the mayor's claims are supportable.
Read the worked answer
The claims are not supportable as causal. Participants self-selected, so motivation, prior employment, and local labor markets differ from the comparison. The comparison group includes retirees and part-time workers, making 61% a poor counterfactual. The 40% survey response and 300 dropouts create attrition bias; completers are likely more job-ready. Six-month follow-up is short. The youth subgroup (n=85) is too small for a stable estimate and shows a much lower rate. The $1.30 wage difference may reflect job mix, hours, or geography. A press release claiming a '7-point gain' overstates the design. It is fair to say completers had higher employment than the citywide rate, while noting the limits.
2. Propose a stronger evaluation design and state what should be reported now.
Read the worked answer
Stronger design. Use a waitlist or randomized rollout among eligible applicants. If randomization is infeasible, use propensity-score matching with pre-registered covariates and a difference-in-differences design using pre-program earnings. Rely on administrative wage records rather than a 40% survey. Follow up at 12 and 24 months. Pre-specify subgroup analyses, including youth, with minimum sample thresholds. Report cost per completer and cost per employed participant.
Report now. Give completer employment and wage outcomes with confidence intervals, the response rate, and a clear non-causal caveat. Compare providers only if sample sizes allow. Use the $4,800 cost to frame value: if the wage premium is real, calculate break-even months. Recommend continuing with a stronger evaluation rather than expanding on the current evidence.
3. How should the council use the evidence to decide among expansion, continuation, and redesign?
Read the worked answer
Treat the decision as a portfolio choice under uncertainty. Expansion is not justified because the causal signal is weak. Continuation with a stronger evaluation is defensible because the program shows promising descriptive outcomes and serves a population with barriers. Redesign is warranted for the youth component, where n=85 and a 54% employment rate suggest the model may not fit younger participants. Allocate the $600,000 to continuation plus a randomized waitlist study, and require provider-level reporting by service model. Set decision rules in advance: expand only if the evaluation shows a statistically credible employment or earnings gain that exceeds cost; redesign or stop if effects are null or negative. Publish the rules so the choice is not seen as political.