Evaluation: designs, validity threats, and economic analysis
Evaluation judges whether a programme was delivered as intended and whether it produced change. Items test design choice, threats to validity, and the difference between process, impact, and outcome measures.
Core rules
- The CDC framework runs: engage stakeholders, describe the programme, focus the design, gather credible evidence, justify conclusions, ensure use and share lessons.
- Formative evaluation tests materials and procedures before launch; process measures delivery; impact measures short-term change in behaviour or determinants; outcome measures health status; summative renders an overall judgement.
- Designs: randomised controlled trial (strongest internal validity), quasi-experimental options (nonequivalent control group, interrupted time series, regression discontinuity), and single-group pre/post, which has no counterfactual and supports only weak claims.
- Threats to validity: history, maturation, testing effects, instrumentation change, regression to the mean, selection, attrition, and contamination between groups.
- Measures need reliability, validity, and sensitivity to change; a tool that cannot detect movement will make an effective programme look inert.
- Economic analysis: cost-benefit (both sides in money), cost-effectiveness (cost per unit of outcome, such as per case prevented), cost-utility (cost per quality-adjusted life year).
- Utilisation-focused evaluation designs around the intended users and their intended uses.
Worked example
A worksite programme reports mean BMI falling 1.2 kg/m² from pre to post with no comparison group. The claim 'the programme reduced obesity' is unsupported: participants often enrol after an unusually high reading (regression to the mean), and seasonal or secular trends (history, maturation) also produce movement. Defensible next step: add a comparison worksite, or run an interrupted time series on existing annual screening data to see whether the change exceeds the pre-existing trend.
Traps that cost marks
- Reporting satisfaction scores as evidence of impact.
- Presenting an attendance count as an outcome finding.
- Post-test-only measurement with no baseline.
- Ignoring attrition, which can make remaining participants look like successes.
- Reporting a p-value while omitting effect size and confidence interval.