Reading a 12-week diabetes trial: confidence intervals, attrition and overclaiming
Read the following abstract.
"We randomly assigned 480 adults with type 2 diabetes (HbA1c 7.5–10.0%) to 12 weeks of oral semaglutide (n = 240) or placebo (n = 240) in a double-blind design. Mean baseline HbA1c was 8.4% in both groups. At week 12, mean HbA1c fell by 0.8 percentage points with semaglutide and by 0.3 points with placebo; the between-group difference was 0.5 points (95% CI 0.2 to 0.8; p = 0.02). The proportion of participants reaching HbA1c below 7.0% was 42% versus 28%. Gastrointestinal adverse events led to discontinuation in 18% and 9% of participants, respectively. An open-label extension to 52 weeks retained 71% of participants; the HbA1c difference at 52 weeks was 0.3 points (95% CI −0.1 to 0.7). The authors conclude that oral semaglutide is effective and well tolerated for long-term glycemic control."
The clinician-defined minimal clinically important difference (MCID) for HbA1c is 0.5 percentage points.
1. Interpret the primary result. Is the effect statistically significant, is it clinically meaningful, and what do the confidence limits add beyond the p-value?
Read the worked answer
Statistical significance. p = 0.02 is below the conventional 0.05 threshold, and the 95% CI for the difference (0.2 to 0.8 points) excludes 0. So the superiority of semaglutide over placebo at 12 weeks is unlikely to be explained by chance alone.
Clinical meaningfulness - the key distinction. The point estimate, 0.5 points, sits exactly at the MCID. But statistical significance and clinical relevance are different questions: the lower confidence limit is 0.2 points, well below the MCID. The data are therefore compatible with an effect ranging from trivial (0.2) to clearly worthwhile (0.8). The p-value tells us only that the true effect is probably not zero; the CI tells us how precisely we know its size, and here precision is modest despite n = 480, because the within-group variability of HbA1c is large.
Absolute figures. The proportion reaching HbA1c < 7.0% rose from 28% to 42%, an absolute increase of 14 points, so NNT = 1/0.14 ≈ 7 (8 to be safe). Conversely, discontinuation for GI events rose from 9% to 18%: absolute increase 9 points, NNH ≈ 11. Roughly one extra patient gains target control for every 1.5 patients who stop the drug - a trade-off the abstract's "well tolerated" phrasing obscures.
2. A colleague states: "p = 0.02 means there is a 2% probability the drug does not work." Identify the error, then evaluate whether the 52-week extension supports the authors' long-term claim.
Read the worked answer
The p-value error. p = 0.02 is P(data at least this extreme | null hypothesis true), not P(null hypothesis true | data). It says that if the drug had no effect, a difference this large would arise 2% of the time. It says nothing directly about the probability that the drug works. Giving the drug a 98% chance of being effective is the classic inversion of conditional probability - the same base-rate fallacy that inflates the perceived value of a screening test.
The 52-week extension does not support the claim.
- The CI crosses zero (−0.1 to 0.7): the difference is no longer statistically distinguishable from no effect, and the point estimate has halved to 0.3, below the MCID.
- Attrition of 29% - nearly a third of participants left; those who remain in an open-label extension are typically the ones who tolerate and benefit from the drug, which biases the estimate upward, not downward.
- Loss of blinding removes protection against ascertainment and co-intervention biases (diet, other glucose-lowering drugs).
- No pre-specified primary endpoint for the extension; a secondary, underpowered analysis cannot carry the weight of a confirmatory claim.
A defensible conclusion would be: "effective at 12 weeks, with a confidence interval compatible with a modest-to-moderate effect and a clinically important excess of GI discontinuation; long-term durability is not established."
3. The sponsor proposes to repeat the study with 1 500 participants per arm and the same 12-week design. Would that fix the main problem? Justify your answer using the relationship between sample size and confidence-interval width, and propose two design changes that would address the real weaknesses.
Read the worked answer
What a larger sample does. The width of a confidence interval for a difference of means scales as 1/√n (standard error = s/√n). Multiplying n by about 6 (240 → 1 500) shrinks the standard error by a factor of √6 ≈ 2.45: the CI would narrow from roughly ±0.3 to ±0.12 points. That would establish precision - the effect could then be shown to exceed, or fall short of, the MCID with confidence.
What it does not fix. Power is not the weakness here; the weaknesses are:
- Short duration - 12 weeks cannot assess durability, and the 52-week data already suggest waning of effect; a larger short trial would answer the wrong question more precisely.
- Tolerability - 18% versus 9% discontinuation is a genuine harm that more power does not mitigate.
- No active comparator - superiority to placebo does not inform the choice between semaglutide and an existing drug.
- Surrogate endpoint - HbA1c is a surrogate; hard outcomes (retinopathy, renal events, cardiovascular events) are what matter clinically.
Two design changes.
- Active-comparator, non-inferiority or superiority trial against the current standard of care, with a pre-registered MCID margin and HbA1c as a secondary endpoint alongside microvascular outcomes.
- 52-week, double-blind, intention-to-treat analysis with a pre-specified handling of missing data (multiple imputation or mixed model for repeated measures), reporting discontinuation and adverse events as co-primary outcomes, so that durability and tolerability are quantified rather than assumed.