Sign-Off on a 41% Attrition Rate
A funded worksite wellness program delivered a 12-week walking challenge to 340 employees across 8 employers. Of those, 199 completed the follow-up survey (41% attrition); completers had higher baseline step counts than non-completers. Results: +620 steps/day (p = .02), body mass index change of −0.1 (p = .41), and a 0.3-point gain on a 5-point self-efficacy scale (p = .03). Two employers withdrew mid-program, citing shift schedules. The funder's annual report is due in 10 days, and the communications director has drafted the headline “Participants increased physical activity by 30% - program proven to work.” You are the evaluator asked to sign off on the draft.
1. Is the 30% claim and the phrase “proven to work” defensible? What must be verified before anything is published?
Read the worked answer
Neither is defensible as written.
Verify first:
- Denominator. 620 steps is roughly 30% only if baseline averaged near 2,070 steps/day. Confirm the baseline mean and confirm the 620 is an absolute mean change, not a percent among a subgroup.
- Whose change it is. Is the p-value from a paired test on the 199 completers? If so, the 30% describes completers, not the 340 enrolled - and completers started more active, so the figure is almost certainly optimistic for the full group.
- Measurement. Pedometer, accelerometer, or self-report? Device wear and reactivity inflate early-program counts.
Why “proven” fails: a single-group pre/post design cannot support causal language. Regression to the mean, seasonality (a spring walking challenge climbs with the weather), history effects, and differential attrition all remain live alternatives.
Also note the mismatch between statistical and practical significance: p = .02 with a small effect size (d ≈ 0.2) is not the same as a meaningful health improvement, while the null BMI result (p = .41) should be reported, not buried.
Recommended action: return the draft with tracked changes, supply the correct descriptive sentence with a confidence interval, and require the report to describe the design as pre/post without a comparison group.
2. How should attrition and the two employer withdrawals be handled analytically and in the report?
Read the worked answer
Analytically:
- Build a completer vs. non-completer comparison table on every baseline variable (steps, BMI, self-efficacy, shift type, employer). This is the evidence base for judging the direction of bias.
- Report both a complete-case analysis and an intention-to-treat analysis (mixed model using all available observations, or baseline-carried-forward as a conservative sensitivity check).
- Run a bounding sensitivity analysis: if non-completers averaged zero change, what is the enrolled-group effect? Report that range, not the completers-only point estimate.
Substantively: the two employer withdrawals are not merely missing data - they are a delivery finding. Shift-schedule barriers mean the program systematically excluded hourly and shift workers, who are often the employees with the highest chronic disease risk. That belongs in the report as a limitation and as the central recommendation for the next cycle: shorter sessions repeated across shifts, paid time, on-site delivery.
In the report: state reach (340 enrolled of how many eligible), dose (mean sessions attended), retention (199/340 = 59%), and the equity implication of who dropped out. Do not present 59% retention as adequate without comment; it caps how confidently the funder can be told the model works.
3. Rewrite the key finding in two sentences and name the three most serious threats to internal validity.
Read the worked answer
Rewrite: “Among the 199 employees who completed follow-up (59% of enrollees), mean daily steps increased by 620 (95% CI to be inserted), a small improvement; body mass index did not change. Because the design had no comparison group and retention was partial, these results are preliminary and do not establish that the program caused the change.”
Three most serious threats:
- Selection/regression to the mean - completers were already more active at baseline, so part of the gain reflects who stayed, not what the program did.
- History and seasonality - a 12-week walking challenge overlapping better weather will show gains even with no program.
- Differential attrition - 41% loss concentrated among shift workers biases the estimate upward and narrows the population the finding applies to.
Lesser threats worth one line each: measurement reactivity from device wear, and testing effects on the self-efficacy scale.