The Word "Significant" in a Public-Health Brief
A public-health brief reports: "Screening rates in the intervention region rose significantly compared with the control region (p = 0.04). The effect was modest but meaningful: a 1.3 percentage-point increase. We therefore recommend national scale-up." A reviewer objects that "significant" and "meaningful" are doing different work in the same paragraph, and that the recommendation does not follow from a 1.3-point change. The brief's authors respond that p = 0.04 is below the conventional 0.05 threshold, so the result is significant, and that "meaningful" simply reflects their judgment that any increase in screening is worthwhile. The study enrolled about 40,000 people per region.
1. Explain the two senses of "significant" at work here and why the paragraph's use of both "significantly" and "meaningful" is confusing to a reader.
Read the worked answer
Statistical sense: "Significantly" means the observed difference is unlikely to have arisen by chance alone under the null hypothesis - here, p = 0.04, below the 0.05 convention. It says nothing about size.
Practical sense: "Meaningful" (or "significant" in everyday usage) means the effect is large enough to matter for policy or health. That is a judgment about magnitude and consequence.
The paragraph uses "significantly" statistically and then "meaningful" practically, in adjacent sentences, without signaling the switch. A reader can easily read "significantly" as "substantially" and conclude the 1.3-point change was large - which it is not. The confusion is compounded because the authors' defense of "meaningful" is purely subjective ("any increase is worthwhile"), which is a values claim, not a finding. The fix is lexical discipline: "statistically significant" for the p-value, and a separate, argued statement about whether 1.3 points justifies national scale-up. The two words should not be allowed to blur into each other.
2. With roughly 40,000 people per region, what does p = 0.04 tell you about the size of the effect? Explain the relationship between sample size, p-values, and practical importance.
Read the worked answer
With very large samples, p-values become sensitive to tiny differences: even a trivial effect will produce a small p-value because the standard error shrinks as sample size grows. So p = 0.04 here tells us the 1.3-point difference is probably not chance - but it tells us almost nothing about whether 1.3 points is worth acting on.
The logic: a p-value combines effect size and precision. With 40,000 per arm, precision is high, so a small effect can still clear the threshold. In a study of 200 people, the same 1.3-point difference would likely not reach p = 0.04, even though the underlying effect would be identical. This is why statistical significance cannot substitute for a judgment about magnitude.
A well-framed answer notes that the authors should report the confidence interval, not just the p-value, so readers can see the plausible range of the effect. If the interval runs from, say, 0.1 to 2.5 points, the reader learns the effect could be negligible or modest - information the p-value alone conceals.
3. Does the recommendation for national scale-up follow from the result? Explain what additional considerations bear on it.
Read the worked answer
It does not follow directly. The result supports a narrow claim: the intervention raised screening by about 1.3 points in this setting, and the difference is unlikely to be due to chance. National scale-up requires more:
- Magnitude versus cost. A 1.3-point gain may or may not justify the program's cost at national scale; that is a budget and priority question, not a statistical one.
- Generalizability. The intervention worked in one region against one control. Effects often shrink when programs scale, because implementation quality varies.
- Baseline rates. A 1.3-point gain where baseline screening is low may matter more than the same gain where it is already high.
- Equity. National rollout may widen or narrow gaps depending on which subgroups respond.
- Confidence interval and durability. Whether the effect persists past the study period is unknown from the brief.
A careful reader treats the recommendation as a policy judgment dressed as a study conclusion. The study licenses "this intervention appears to work modestly in this setting"; it does not license "scale nationally." The authors' own defense - that any increase is worthwhile - is a value premise they should state openly rather than fold into the findings.