Designing an Evaluation for a Salt-Reduction Policy Under Political Pressure
A ministry of health plans to mandate a 20% reduction in the sodium content of bread and processed cheese over 3 years, with voluntary reformulation in year 1 and mandatory limits from year 2. Baseline mean population sodium intake is estimated at 4,300 mg/day from a 24-hour urinary sodium survey (n=600). Hypertension prevalence is 31% in adults. The minister wants a result 'within 18 months' to justify the policy before a budget review, and the food industry has offered to co-fund the evaluation and to provide its own sales and reformulation data. There is no national dietary survey and no food composition database. The evaluation budget is modest.
1. Identify the main threats to a credible evaluation and how to mitigate them, given the timeline and the industry offer.
Read the worked answer
Threats
- Timeline vs biology: 18 months is too short to detect a change in mean population sodium intake with confidence, and far too short to detect a change in hypertension prevalence or cardiovascular events. Promising outcome results at 18 months would be unsupported; promising them and failing would damage the policy.
- Industry co-funding and data: creates a conflict of interest and gives the funder control over the very numbers that measure compliance. Industry sales data are useful but must be independently verified.
- No dietary survey or food composition database: you cannot measure reformulation or substitution without them, and you cannot detect whether sodium was removed from bread but added to other products.
- Secular trends and co-interventions: other policies (labelling, taxation) may change simultaneously; without a comparison you cannot attribute change.
- Measurement error: a single 24-hour urine is subject to day-to-day variability; n=600 gives limited power to detect small changes.
Mitigations
- Pre-specify a process-and-intermediate-outcome evaluation for 18 months (reformulation compliance, sodium content of key foods, population intake modelled from food composition), and reserve outcome evaluation (urinary sodium, BP) for 3-5 years.
- Independent evaluation team; industry data accepted only if auditable and cross-checked with retail sampling.
- Build a food composition database from retail sampling of the top-selling products before and during implementation.
- Use a comparison jurisdiction without the mandate where feasible, or interrupted time-series with a clear pre-period.
2. Propose the indicator set you would report at 18 months and at 5 years, and explain how you would answer the minister's demand for a result now.
Read the worked answer
At 18 months (process and intermediate)
- Proportion of bread and cheese products meeting the sodium targets (retail sampling, not industry self-report).
- Mean sodium content of the top 20 bread and cheese products.
- Modelled change in population sodium intake from food composition and sales data.
- Industry compliance rate; number of products reformulated versus reformulated with compensating changes elsewhere (e.g., sodium moved to other categories).
- Consumer awareness and any evidence of substitution to higher-salt products.
At 5 years (outcome)
- Mean 24-hour urinary sodium (repeat survey with adequate power).
- Hypertension prevalence and mean systolic BP.
- Cardiovascular event rates where data permit.
- Equity analysis by income group (reformulation can widen or narrow gaps).
Answering the minister
- Be direct: a biological result in 18 months is not achievable, and claiming one would be unscientific. What is achievable in 18 months is credible evidence that the policy is being implemented and that the food supply is changing - which is what a budget review actually needs to decide whether to continue.
- Frame the 18-month report as a fidelity and early-effect report, with the 5-year report as the outcome evaluation.
- Refuse industry co-funding of the evaluation; accept data under audit.
- Recommend starting the food composition database and the baseline retail sampling immediately, as these are the rate-limiting steps.