The Subgroup That Saved the Launch
You lead a product test for a beverage launch: n=1,000, nationally recruited, with a pre-registered analysis plan naming full-sample purchase intent as the primary endpoint. Results arrive. Full-sample intent is 22% against a category benchmark of 30%. Among a pre-specified but secondary segment - "heavy flavour-seekers," n=140 - intent is 38%, comfortably above benchmark. The client's CMO emails: "This is great. Headline the 38% among our target consumers, and drop the two aftertaste questions - they're confusing." A colleague notes that four other subgroups were also examined, one of which came in at 12%, and that the aftertaste questions are the only measures of a known formulation problem. The launch decision meeting is in nine days and the client's team has already briefed retail partners on a strong result.
1. How do you respond to the CMO's reporting request?
Read the worked answer
Agree on what is true and refuse what is not. The 38% among heavy flavour-seekers is a real finding and can be reported - as a secondary, subgroup result with its n of 140, its confidence interval, and the fact that it was one of five subgroups examined. It must not be the headline, because the pre-registered primary endpoint is full-sample intent at 22% versus a 30% benchmark, and that is the result the study was designed to test.
The distinction is not pedantry. With five subgroups, one above-benchmark result is close to what chance alone would produce, so presenting it as the headline converts a fishing expedition into a launch case. The correct framing is: full sample below benchmark, with a segment that responds positively and a segment that responds poorly; the commercial question is whether the flavour-seeker segment is large enough to sustain the launch.
On dropping the aftertaste questions: refuse. Removing measures of a known formulation problem from the report because they are "confusing" is suppression of adverse findings. Offer instead to present them clearly - a short slide, plain language, with the finding and its implication. If the client wants to omit them from a public summary, say that the internal report will contain them and that you will not sign a report that does not.
2. How do you present the subgroup results without overclaiming or hiding them?
Read the worked answer
Use a fixed structure. First, the primary endpoint as pre-registered, with base, estimate and interval. Second, the pre-specified secondary segments, each with its base and interval, flagged as exploratory if they were not part of a multiplicity-controlled plan. Third, a clear statement of how many subgroups were examined, so the reader can judge the risk of a chance finding. Fourth, the adverse signals, placed alongside the positive ones rather than in an appendix.
Where a subgroup is small, say so explicitly: n=140 means the interval around 38% is wide, and the report should give it. If the segment is commercially important, recommend a confirmatory study powered for that segment rather than treating an exploratory result as established.
Avoid two symmetrical errors: presenting 38% as the finding, and burying the subgroup entirely because the primary endpoint failed. The client needs to know that a flavour-seeker segment exists and responds; that is genuinely useful. They also need to know it is not the same as a national launch case. The report should make both readable in the executive summary, in that order of emphasis.
3. What do you document, and what do you do if the client publishes the 38% figure without context?
Read the worked answer
Document contemporaneously: the pre-registered analysis plan with its date, the full results file, the email exchange about the headline and the aftertaste questions, your written recommendation, and the final report as delivered. Keep the version history. If the client later publishes a claim you advised against, that record is what distinguishes your conduct from theirs.
Before publication, ask to see any external summary or press release that cites the study and offer to check it. If the client publishes "38% purchase intent" without the base or the primary result, respond in writing and promptly: a letter to the CMO setting out what the study found, what the published figure omits, and asking for correction. Escalate internally to your firm's ethics or legal lead, and consider whether your firm's name may appear on the material - if it does, you have a stronger basis to demand correction or withdrawal of the attribution.
If correction is refused and your firm is named, withdrawing permission to cite the study is a legitimate last step, taken in writing. It is uncomfortable and commercially costly, but the alternative is being the author of record for a claim you know to be misleading.