A Supplement Trial With No Placebo: Reading the Confidence Interval Correctly

Standard error of the mean change:

• SEM = 24 ÷ √20 = 24 ÷ 4.47 ≈ 5.4 ms

95% CI:

• 13 ± 1.96 × 5.4 = 13 ± 10.5 → 2.5 to 23.5 ms faster

The interval excludes zero, so the observed improvement is unlikely to be explained by random sampling variation alone (P < 0.05). That is a statement about precision, not about validity.

Why it does not prove the supplement works:

1. No placebo arm - expectation and the ritual of taking a pill can produce arousal effects of this size.
2. Only one discarded practice trial - reaction time improves with familiarization between blocks; the second block is always faster regardless of any pill.
3. No blinding - the assistant knew the assignment and could cue participants.
4. Unblinded stimulant - 80 mg of caffeine is a pharmacologically active dose, so any genuine effect cannot be attributed to the "focus" ingredients.
5. Time-of-day and single session - 9 a.m. alertness rises on its own.

Judgment: the data are statistically distinguishable from zero but the design cannot assign causation. A significant CI from a confounded design is a precise estimate of the wrong quantity.

Design: randomized, double-blind, placebo-controlled crossover.

• Placebo capsule (identical appearance and taste): removes expectancy and ritual effects - the largest single threat here.
• Randomization: removes selection/allocation bias (e.g., faster students volunteering first).
• Blinding of both participant and test assistant: removes cueing and scoring bias.
• Crossover with counterbalanced order and ≥48 h washout: every student serves as their own control, removing between-person variability and balancing order/practice effects across arms.
• Multiple baseline trials until performance plateaus (e.g., 5 trials, use the last 3): removes the familiarization effect instead of trying to discard it with one trial.
• Caffeine matched or absent in both arms: removes the confound; if caffeine is the active ingredient, that is the claim to test.
• Pre-registered primary outcome and sample-size calculation: prevents post-hoc selection of whichever measure looked best; with SD of changes = 24 ms and a smallest meaningful effect of 10 ms, the study needs roughly 60 participants per arm for 80% power, not 20.
• Standardized testing time and environment: removes circadian and noise variation.

Point estimate: supplement advantage = −13 − (−10) = −3 ms (3 ms faster, favoring supplement).

Interval: 95% CI −7 to +13 ms, which includes zero. (Sign convention here: negative favors the supplement, so the interval spans from a 7 ms benefit to a 13 ms slower performance.)

Correct conclusion: there is no evidence of a benefit. The study cannot rule out a trivial benefit of a few milliseconds, but it rules out - with 95% confidence - any benefit larger than about 7 ms.

Context matters for the interpretation: the original unblinded comparison produced a 13 ms "effect," and roughly 10 of those 13 ms were reproduced by the placebo. That is the signature of expectancy and practice, not pharmacology.

How to report it:

• State the estimate with the interval, not P alone: "3 ms favoring supplement, 95% CI −7 to +13 ms."
• State the smallest effect that would matter to a student (probably ≥10–15 ms, i.e., a real change in test performance) and note the interval is almost entirely below that threshold.
• Do not claim the supplement "failed" or "worked"; say the trial did not detect a meaningful effect and give the interval so readers can judge.
• Disclose the caffeine content and the change in design, so the earlier result can be understood as a confounded preliminary study.