When the Data Looks Too Good
You run a 6-session financial education program and report outcomes to a city funder. Of 200 enrollees, 140 attend at least one session, 90 complete all six, and 70 submit both pre- and post-assessments. Mean post-test scores are 18 points higher than pre-test. Your development director drafts a report stating '200 residents improved their financial knowledge by an average of 18 points.' She argues that reporting only the 70 who completed both tests 'undersells our impact' and could cost renewal funding. You also notice that completers were disproportionately those referred by a partner agency that pre-screens for readiness.
1. What is wrong with the draft claim, and what is the accurate way to describe the 18-point result?
Read the worked answer
The draft claim is wrong on three counts. First, it attributes the result to 200 residents when only 70 provided both pre- and post-assessments; 130 either did not complete or did not submit both, so no change can be computed for them. Second, 'improved' asserts causation and individual change from a group mean difference; without individual matching and a comparison, the 18-point mean gap is a descriptive statistic, not proof each person improved. Third, it omits selection: the 70 are a self-selected and partner-screened subset, likely more ready to learn, so the result does not generalize to all 200.
Accurate framing: 'Among the 70 participants who completed both a pre- and post-assessment, mean scores were 18 points higher at post-test. These participants were a subset of 200 enrollees and may differ from those who did not complete both assessments, so results should not be generalized to all enrollees.' Report completion and assessment-response rates alongside the score change.
2. The development director says accurate reporting risks renewal funding. How do you resolve the tension, and what would you add to the report to strengthen it honestly?
Read the worked answer
Resolve it by recognizing that inflated claims are the bigger renewal risk: a funder who discovers that '200 improved' really means '70 tested' may terminate the relationship and damage your reputation. Accuracy and sustainability point the same direction once you report the full picture.
Strengthen the report honestly by adding: (1) the full participation funnel - 200 enrolled, 140 attended, 90 completed, 70 with matched assessments - so the funder sees engagement, not just a headline; (2) the 18-point result properly qualified, with the selection caveat; (3) qualitative or behavioral indicators (e.g., completion of a budget or savings action) that capture value the test may miss; (4) an improvement plan naming the specific retention and assessment-response problems and how you will address them next cycle. This converts a weak-looking number into a credible story of measured progress and honest learning, which is what sophisticated funders actually reward.