Data Analytics and Evidence-Based Decision-Making in Government
This domain tests whether you can move from a decision-relevant question to a defensible answer: choosing a design, stating the counterfactual, and reporting uncertainty honestly.
Core rules and concepts
- Begin with the decision the evidence must inform, then build a logic model and select measures that trace activities to outcomes.
- Separate descriptive, predictive, and causal claims. Only a design with a credible counterfactual supports "X caused Y".
- Design options: randomised controlled trial (strongest counterfactual), difference-in-differences (requires parallel trends), regression discontinuity (requires an assignment cutoff), interrupted time series (requires adequate pre-period points), and matching (controls only for observed variables).
- Named threats: selection bias, attrition, secular trends, regression to the mean, spillovers, and changes in measurement definitions.
- Report effect size and interval, not just significance, and pair effects with cost to support cost-effectiveness judgements.
- Publish a learning agenda so that unanswered questions are scheduled rather than discovered late.
Worked example
A training programme is rolled out in selected offices. Treated offices improve their placement rate by 12 points; comparable untreated offices improve by 5 points. The naive pre/post gain is 12 points, but the difference-in-differences estimate is 12 − 5 = 7 points. Reasoning: the comparison group captures the common trend, so only the excess gain is attributable. If the treated offices were chosen because they were already improving, the parallel-trends assumption fails and 7 points overstates the effect - check pre-period trends or strengthen the design.
Traps that cost marks
Reading a pre/post change as causal; selecting a baseline that flatters the result; confusing statistical with practical significance; ignoring differential attrition; reporting a point estimate with no uncertainty range.