APCAT study materials
APCAT study resources

Seventy-Six Percent, If You Drop the Right People

APCAT · Put your knowledge to work in context.

All case studies

Seventy-Six Percent, If You Drop the Right People

A funder's grant renewal depends on demonstrating that at least 75% of participants met the program's primary outcome. Your 12-month evaluation, using the pre-registered definition, shows 58% (29 of 50) met it. The executive director points out that if you exclude the 9 participants who attended fewer than 60% of sessions - "they weren't really in the program" - the rate becomes 71%; excluding two more who moved away gives 76%. She asks you to present "76% of engaged participants" in the renewal report and to drop the per-protocol caveat because "the funder won't read footnotes." You are the only person who has seen the full dataset, and your contract renews in three months.

1. Identify the ethical issues in this request and the principles that bear on them.

Read the worked answer

This is a request to mislead a funder, and the fact that a version of the number is defensible makes it more dangerous, not less.

  • Selective reporting. 76% appears only after two post-hoc exclusions chosen because they push the figure over a threshold. 58% was the pre-specified analysis; the others are exploratory. Headlining the exploratory number inverts the hierarchy of evidence.
  • Material omission as misrepresentation. The per-protocol caveat is the one sentence that lets a reader interpret the number correctly. "They won't read footnotes" is an argument for making the method more prominent, not for deleting it.
  • Conflict of interest. Your contract renews in three months. That pressure has to be disclosed and managed, not silently absorbed - and the same is true for the director, whose job depends on the renewal.
  • Power differential. You are the only person who can check the number. Being asked privately to shade it is exactly the situation where a professional obligation is doing its work.
  • Duty to participants and to the funder. These data describe real people's outcomes. Distorting them corrupts a funding decision and, if published, the evidence base other programs rely on.

2. Draft the substance of your response to the director.

Read the worked answer

Say no to the framing, yes to more information - and offer something she can defend.

The response in substance:

I can give the funder a stronger report than the one you've described, but I can't headline 76%. Here is what I can do:

  • Primary result, pre-specified, all enrolled: 58% (29/50). This answers the question the grant actually asked.
  • Secondary, clearly labeled as post-hoc: 71% (29/41) among participants attending ≥60% of sessions. That is a legitimate and useful finding - it says dose matters - and it belongs in the report with the exclusion rule stated in the body, not a footnote.
  • Adding the two movers to reach 76% (29/38) is not defensible. At that point the exclusion rule is whatever gets us over the line, and a funder's analyst will see it in about thirty seconds.
  • Keep the limitations section, one page, plain language, up front. Reviewers reward candor about what didn't work and what changes next year; they punish a number that collapses under one question.

I'll draft it this way today and you can review before it goes anywhere.

The practical argument matters as much as the ethical one: a report built on an indefensible number exposes the organization to audit findings and lost credibility, and it is a weaker renewal case than an honest report with a dose-response story.

3. What do you document, and what do you do if the director overrides you?

Read the worked answer
  • Document contemporaneously. The request as made, the options you presented, the version of the report you recommended, and the final version issued with who approved it. Keep dated copies of the analysis file and the correspondence.
  • If she overrides you: put your objection in writing to her, restate in one paragraph why 76% cannot be supported, and offer the alternative once more. Ask that your name and analysis not be attached to the misleading claim - that the report not be issued over your signature or described as your evaluation.
  • Escalate if it is issued anyway: the organization's compliance or audit function, then the board, then the funder's program officer in writing. Check whether your certifying body imposes its own reporting obligation - several do.
  • Never alter the underlying data, backdate an analysis decision, or write a methods note describing an exclusion rule you did not pre-specify. That converts a reporting dispute into research misconduct.
  • Consider withdrawing if the pattern continues. A contract that requires misrepresentation is one you cannot perform competently, and staying while it happens makes you a participant rather than a bystander.