The 87% Claim in the Grant Application
A nonprofit's flagship program is a four-week texting-based financial boot camp. Last year 2,400 people enrolled and 1,110 completed at least three of the four modules. Everyone who finishes receives a survey; 344 respond. Of those, 87% agree with 'I feel more in control of my money,' and 62% answer yes to 'did your credit score improve?' - with a self-reported average gain of 31 points among those who supplied a number. There is no comparison group, and participants enroll themselves, often after a marketing push on a local radio station. The credit score item is free text and several answers are ranges ('up maybe 20 to 40'). A regional credit bureau has offered de-identified score data, but only for members who opt in, and the data-sharing agreement takes about six weeks. The development director has 10 days before a $250,000 foundation proposal is due and wants to write '87% of graduates improved their financial outcomes' and 'average credit score increase of 31 points.' The foundation asks for 'evidence of effectiveness' and explicitly permits preliminary findings.
1. Assess the validity of the two proposed claims. What exactly does the data support?
Read the worked answer
Both claims fail on measurement, not on intent.
The 87% claim.
- Wrong numerator and denominator. 87% is of 344 respondents, not of 1,110 completers (a 31% response rate) and certainly not of 2,400 enrollees. Post-program survey respondents are systematically more engaged than non-respondents; this is ordinary non-response bias, and it inflates every favourable number.
- The construct is confidence, not outcome. 'I feel more in control' is a legitimate early indicator of behaviour change. It is not a financial outcome.
- 'Graduates' is undefined - 1,110 or 2,400? The sentence as written implies the larger base.
The +31 points claim.
- Self-report of a figure people often do not know, with ranges that cannot be averaged without inventing precision.
- No counterfactual. Scores move for reasons unrelated to the program - paying down a card, aging out a delinquency, a credit-repair advertisement. Self-selection plus a radio campaign means participants may already be in a change phase.
- A mean without a distribution. +31 could be fifteen people at +200 with everyone else flat, or a broad small shift. The difference matters enormously to a funder.
- No time frame - improved since when?
Verdict: this is a reasonable pilot feedback instrument. Neither sentence is supportable as written.
2. What can be claimed in the proposal, and what should the team do in the 10 days available?
Read the worked answer
Rewrite at the level the data supports: report denominators, name the instrument, and state the limitation in the same paragraph.
Defensible versions:
- 'Of 344 participants who responded to the post-program survey (31% of 1,110 completers), 87% agreed they felt more in control of their money.'
- 'Among the 214 respondents who reported a credit score change, the average self-reported increase was 31 points. These figures are self-reported and have not yet been verified against bureau data.'
- 'The program has no comparison group, so these results describe participants rather than the program's causal effect.'
Then add the element that actually strengthens a proposal: a credible plan. 'In the coming year we will link program records to de-identified bureau data for consenting participants (agreement in progress, approximately six weeks), compare score changes against a matched group drawn from the waitlist, and pre-register our indicators before the cohort begins.' A program officer reading that understands exactly where you are and where you are going.
Cut entirely: '87% of graduates,' 'improved their financial outcomes' (confidence is not an outcome), and any framing of 31 points as a program effect.
Internal check: have one person who did not run the program read the paragraph and state what it claims. If they hear causation, rewrite it. The honest version takes two hours and is far stronger than an inflated one that a program officer will discount on sight.
3. Design the evaluation the program should run next year, and identify the data-ethics requirements that come with using bureau data.
Read the worked answer
Design.
- Pre/post with a comparison group. The waitlist is a natural comparison: people who enrolled but start next month. Random assignment by cohort is feasible at 2,400 enrollees and would move this from 'promising' to 'evidence.'
- Verified outcome. Bureau data for consenting participants under a written data-sharing agreement, minimum necessary fields (score, date, no account numbers), de-identified before analysis.
- Completion as moderator, not gate. Model outcomes against modules completed. The roughly 54% who dropped out early are the population most worth understanding, not an inconvenient denominator.
- A validated scale for the confidence measure instead of a single agree/disagree item.
- Pre-register the primary outcome and analysis plan before the next cohort, and disaggregate by income band, age, and referral source.
Ethics.
- Consent must be separable. Agreeing to the survey cannot be bundled with agreeing to bureau linkage, and declining linkage must not affect access to the program.
- No coercion through incentives. A modest gift card for a survey is fine; a large payment conditioned on high scores is not.
- Data minimization and retention. Decide the destruction date now, and who may see the linked file.
- Feedback to participants. If their data produced a finding, tell them - it is both ethical and good for retention.
- Outside review. Even where no IRB applies, have someone outside the program read the protocol before it runs.
Framing for the funder: 'we are building a credible evidence base' is more persuasive than a number that will not survive scrutiny - and it is the only version that will still be true in three years.