A Pre/Post With No Comparison Group and a Statewide Campaign in the Middle
A worksite wellness coalition ran an eight-week walking challenge for 240 employees across nine employers. Participants self-reported minutes walked per week at baseline and at program end. Follow-up was collected from 147 people (61%). Mean days walked per week rose from 2.1 to 3.8, and 78% of completers said the program was 'very helpful.' A statewide media campaign promoting walking launched in week 5. There was no comparison group and no pre-registered analysis plan. The funder's template asks for 'outcomes achieved' and the coalition's executive director wants to submit the 2.1 → 3.8 change as evidence the program worked, with a press release.
1. What can and cannot be concluded from these numbers? Name the specific threats and why each one matters here.
Read the worked answer
The design supports a claim of association, not attribution. The 1.7-day increase is real among the 147 people who answered, and that is the limit of the clean claim.
Specific threats, each grounded in this dataset:
- Attrition bias. You lost 93 of 240 (39%). If those who dropped out had lower or unchanged walking - plausible, since disengagement and low activity travel together - the completer mean overstates the program effect. Report the follow-up rate next to the result, always.
- Secular trend. The statewide campaign launched in week 5 and targets exactly this behavior. It is a rival explanation you cannot rule out and did not have a design to isolate.
- Regression to the mean. Participants who enroll in a walking challenge are plausibly people who just had an inactive stretch.
- Social desirability and self-report. End-of-program measurement is taken by the people who want the program to have worked, from participants who liked it. Minutes walked is a heavily over-reported behavior.
- Instrumentation. If the end survey was administered in person at a celebration event and the baseline was online, the measurement changed, not just the behavior.
Reasoning: the director is not wrong that something happened. The error is the inferential leap from 'participants walked more' to 'the program caused it.'
2. You cannot add a control group retroactively. What analyses can you actually run on the existing data to strengthen or weaken the causal story?
Read the worked answer
Cheap analyses that meaningfully move the needle:
- Completers vs. dropouts at baseline. If dropouts' baseline walking was similar to completers', attrition bias is less likely to explain the gain; if dropouts walked more at baseline, they may have dropped because the program added nothing - a different and equally important finding.
- Dose–response. Compare change in days walked across attendance bands (e.g., 1–3, 4–6, 7–8 weeks attended). A monotonic gradient is the single strongest internal check available without a control group; a flat line suggests the change is not program-driven.
- Employer-level variation. Nine employers, plausibly different implementation quality and different exposure to the state campaign. If gains track implementation intensity, that is informative; if every site moved equally in week 5 onward, suspect the campaign.
- Timing check. Ask whether any site has weekly rather than only baseline/end data - a jump coinciding with week 5 is evidence for the secular-trend explanation.
- Sensitivity analysis. Report the primary result under a pessimistic assumption that all 93 non-responders did not change. That bounds the effect honestly.
Reasoning: none of these rescues causal inference, but dose–response and the attrition bound convert an unfalsifiable claim into a calibrated one, which is exactly what an evaluator owes a funder.
3. The executive director still wants the press release. Draft the substance of what you will and will not say, and what the funder report should contain.
Read the worked answer
What I will say: 'Among the 147 employees who completed both surveys, average walking days rose from 2.1 to 3.8 per week - and employees who attended more weeks saw larger gains.'
What I will not say: 'the program increased walking by 1.7 days,' 'the program caused,' or any figure presented without the follow-up rate.
Press release essentials:
- Put the 61% follow-up rate in the same sentence as the result. '61% of participants completed follow-up' is not a caveat to bury - it is the sentence that keeps the release honest.
- Note the concurrent statewide campaign as a possible contributor in plain language, one clause.
- Lead with the reach story and the employer participation, which are factual and uncontested, rather than the effect size.
- Quote a participant, not a statistic, for the human interest angle.
Funder report structure:
- Outputs (factual): 240 enrolled, 9 employers, 147 with follow-up, 78% satisfaction among completers.
- Outcomes (hedged): pre/post change among completers, labeled as preliminary and uncontrolled.
- Limitations (explicit, bulleted): no comparison group, 39% loss to follow-up, self-report, concurrent statewide campaign.
- Recommendation for year two: add a waitlist or delayed-start comparison group, collect weekly step data via app or pedometer rather than recall, and pre-register the analysis before launch. Offering this unprompted is what makes the coalition a credible evaluation partner rather than a vendor.