The Scientist, the Sponsor, and the Missing Data
A researcher completes a trial of a new study-skills intervention across four schools (n = 640). The pre-registered analysis shows no significant improvement in exam performance, but a post hoc subgroup analysis of one school (n = 90) shows a large positive effect. The trial's commercial sponsor, which funded the study, emails the researcher: 'The subgroup finding is the interesting one. Lead with it in the report, and we'd prefer the pre-registered null result appear in an appendix.' The researcher's department head notes that the sponsor has funded three further projects and that 'these relationships take years to build.' The researcher also recalls that the subgroup was not randomly selected and that the school in question had recently changed its teaching methods.
1. Identify the epistemic problem with leading with the subgroup finding, independent of any ethical concern.
Read the worked answer
The subgroup analysis was post hoc and not pre-registered. This creates several problems:
- Multiplicity / selective reporting. Testing many subgroups raises the chance that at least one will show an apparent effect by chance. If the subgroup was chosen after seeing the data, the reported effect size is inflated.
- Loss of the pre-registration guarantee. Pre-registration exists precisely to prevent the hypothesis from being tailored to the data. Presenting a post hoc finding as the headline reverses that safeguard.
- Confounding. The researcher knows the school recently changed its teaching methods, so the subgroup differs from the others in a way unrelated to the intervention. The 'effect' may reflect that change.
- Small n. Ninety participants in a single school is a weak basis for a strong claim.
So the subgroup result is a hypothesis for further testing, not a finding to lead with. Leading with it would misrepresent the evidential strength of the result - a problem even if the sponsor were entirely benign.
2. Does the sponsor's funding relationship create a conflict of interest for the researcher? Distinguish a conflict of interest from actual bias.
Read the worked answer
A conflict of interest exists when a person has a secondary interest (here, continued funding and the department head's concern about 'relationships') that could improperly influence the discharge of a primary obligation (reporting the science accurately). It does not require that influence actually occur.
This case plausibly involves a conflict: the sponsor has funded three further projects, and the department head has explicitly raised the relationship as a consideration. That is a secondary interest capable of distorting judgement.
Actual bias is a further claim - that the researcher's judgement is in fact distorted. We have no evidence of that; the researcher has so far resisted. Conflating the two is a common error: one can have a conflict of interest and act impeccably, and one can be biased without any formal conflict.
The practical upshot is that the conflict should be managed (disclosure, independent review of the report, editorial control protected contractually) rather than treated as proof of wrongdoing. If the researcher does lead with the subgroup finding, the pre-existing conflict becomes relevant context for judging whether the decision was influenced.
3. The department head says 'these relationships take years to build.' How should the researcher weigh this against obligations to the trial participants and the wider field?
Read the worked answer
The department head's point is a genuine institutional consideration - research funding is scarce, and relationships are real assets. But it must be weighed against obligations that are not fungible with it.
To participants: 640 students and their schools consented to take part in a trial whose purpose was to test the intervention. Reporting a misleading headline converts their contribution into support for a claim the data do not establish. Participants have a reasonable expectation that their data will be reported faithfully.
To the field: if the subgroup finding enters the literature as the headline, other schools may adopt an intervention that does not work, and later meta-analyses will be skewed. The cumulative cost of selective reporting is large and falls on people who never agreed to bear it.
Weighing: the funding relationship is a * prudential* consideration about the researcher's future capacity to do research; the reporting obligation is a fiduciary and epistemic one owed to identifiable people. Prudential considerations can shape how a message is delivered (the researcher can explain the null result carefully, offer to design a follow-up study) but cannot justify misrepresenting the evidence. A useful test: would the department head's reasoning be acceptable if stated openly in the report's methods section?