The Outlier That Changed the Conclusion
A student measures the effect of a fertiliser on seedling height. She has 12 seedlings per treatment (fertiliser vs control) and records heights after four weeks. One control seedling is 4.1 cm while the rest of the control group ranges from 8.2 to 9.6 cm. She suspects the small seedling was damaged when a shelf collapsed, but she did not record this at the time. When she includes it, the fertiliser group appears significantly taller (p = 0.04). When she removes it, the difference is no longer significant (p = 0.11). She wants to remove the value because 'it is obviously an outlier' and report the non-significant result as showing the fertiliser does not work. Her draft conclusion says: 'Fertiliser has no effect on seedling height.'
1. Was removing the data point defensible? Analyse the decision using principles of data handling and evidence.
Read the worked answer
Removing the point is not automatically defensible, and the way it was done here is problematic for three reasons:
- The decision was outcome-dependent. The student removed the point after seeing that it changed the significance of the result. This is a form of selective analysis that inflates the risk of a false conclusion. A defensible exclusion rule must be set before analysis (e.g. 'exclude seedlings with documented physical damage') and applied consistently to all groups.
- The justification is retrospective and undocumented. The shelf-collapse explanation may be true, but it was not recorded at the time, so it cannot be verified. Anecdotal post hoc explanations are weak evidence for exclusion.
- The point is not statistically extreme enough to justify automatic removal. At 4.1 cm it is far below the control range, but with only 12 per group, formal outlier tests (e.g. Grubbs') have low power and are themselves unreliable in small samples.
The better approach is to report both analyses transparently: the full dataset and the sensitivity analysis excluding the point, and to discuss how the conclusion depends on that single value. That dependence is itself an important finding about the fragility of the result.
2. Rewrite the conclusion so that it accurately reflects the evidence, and explain why your version is more scientifically defensible.
Read the worked answer
Revised conclusion: 'Across 12 seedlings per treatment, fertiliser-treated seedlings were taller on average. However, the difference was not statistically significant when all data were included (p = 0.04 with one anomalous control value; p = 0.11 without it). The anomalous value (4.1 cm, versus 8.2–9.6 cm for other controls) was not documented at the time of measurement, so its exclusion cannot be justified post hoc. The result is therefore inconclusive: the data are consistent with a moderate positive effect but also with no effect. A larger, pre-registered study with documented exclusion criteria is needed.'
Why this is more defensible:
- It reports the full data and the sensitivity analysis, so the reader can judge the fragility.
- It avoids the false dichotomy of 'significant = effect, non-significant = no effect'.
- It distinguishes absence of evidence from evidence of absence.
- It states what would resolve the uncertainty, which is the proper function of a conclusion in a research report.