The Nutrition Program Everyone Loved
A community health center funds an eight-week peer-led nutrition program in six elementary schools. You coordinated the evaluation: 420 fourth and fifth graders completed a pre-test; 256 completed the post-test. Nutrition knowledge rose from 54% to 78% correct. Self-reported fruit and vegetable intake did not change. Satisfaction surveys were returned by 92% of participating classrooms, and 88% of students said they 'loved' the peer leaders. There was no comparison group and no follow-up beyond the last session. The center's director wants to expand to all 12 elementary schools next year and has asked you for 'the evidence that it works.' A school board member cites the satisfaction data as proof.
1. Interpret the results. What claims can and cannot you make?
Read the worked answer
What you can say:
- Among students who completed both tests, nutrition knowledge improved substantially.
- Acceptability was high among students and teachers.
- The program was feasible to deliver in six schools.
What you cannot say:
- That the program changed eating behavior. Self-reported intake did not change, and self-report is vulnerable to social desirability.
- That the program caused the knowledge gain. Without a comparison group, maturation, testing effects, and classroom instruction could explain it.
- That it 'works' in the sense of improving health outcomes.
- That satisfaction is evidence of effectiveness. Liking a peer leader is not the same as changed behavior.
Attrition: Only 61% completed the post-test (256 of 420). The 164 missing students may differ systematically-for example, absentees or less engaged students-so the knowledge gain may be overstated.
Reasoning: Separate three questions: Was it delivered? Did participants like it? Did it change knowledge or behavior? The data answer the first two and only part of the third.
2. Identify the threats to validity and the evaluation design you would recommend next.
Read the worked answer
Threats:
- Selection: no comparison group; schools or classrooms may differ.
- History and maturation: students change over eight weeks regardless of the program.
- Testing: the pre-test may prime answers on the post-test.
- Attrition: 39% missing post-tests.
- Instrumentation: self-reported intake is imprecise; a single question may not capture change.
- Social desirability: students may report what peer leaders want.
- Contamination: comparison students in the same schools could learn from participants.
Recommended design:
- A quasi-experimental design with matched comparison schools, or staggered rollout (step-wedge) so all schools eventually receive the program.
- Validated measures such as a brief food-frequency or 24-hour recall, plus a knowledge and self-efficacy scale.
- Follow-up at 3 months, not immediately after the last session.
- Implementation fidelity and dose tracking, because null behavior results may reflect weak delivery.
- Pre-specified analysis and sample size.
Reasoning: The current design can describe but not attribute. A comparison condition and follow-up are the minimum for a causal claim.
3. How would you communicate this to the director and the board member without overclaiming?
Read the worked answer
Write a two-page brief with four sections: the question, the design, the findings, and the limits and next steps. Use a simple table: knowledge improved; behavior did not change; satisfaction was high; 61% completed the post-test; no comparison group.
To the director: Say, 'We have evidence of knowledge gain and strong acceptability, but not yet evidence of behavior change. I recommend a staged expansion with a comparison design rather than district-wide rollout.' Offer a pilot in six additional schools with matched comparisons.
To the board member: Acknowledge the satisfaction data as real but explain that satisfaction measures whether students liked the program, not whether it changed what they eat. Avoid jargon such as 'internal validity'; say, 'We cannot yet rule out other explanations.'
Reasoning: Stakeholders hear 'evidence' as 'proof.' The evaluator's job is to protect the decision from an unsupported claim while preserving enthusiasm. Recommending a staged expansion gives the director a path forward without overstating the data.