Analysing a student experiment's statistical claim
A student compares two irrigation regimes on tomato yield: drip (n = 4 plots) and furrow (n = 4 plots). Drip mean = 48.2 t/ha, SD = 3.1; furrow mean = 44.0 t/ha, SD = 4.4. The student runs a t-test and reports p = 0.04, then writes: 'Drip irrigation significantly increased yield by 4.2 t/ha (p < 0.05), proving drip is the better system for all tomato growers.' The plots were not randomised: the drip plots were the four closest to the shed, and the furrow plots were at the far end of the block. Soil depth decreases toward the far end.
1. Critique the student's statistical and causal claims.
Read the worked answer
The statistical result and the causal claim need to be separated.
- The test itself: a t-test on n = 4 per group is possible, but the assumptions are fragile: small samples, unknown normality, and the SDs differ (3.1 vs 4.4), which may violate equal-variance assumptions. A p of 0.04 is marginal and with such small samples is sensitive to a single plot. The result should be reported with the means, SDs, n and a confidence interval, not just 'p < 0.05'.
- Non-random allocation: the drip plots are near the shed and the furrow plots at the far end, where soil depth decreases. Treatment is therefore confounded with soil depth and position. Any difference in yield could be due to soil, not irrigation. Randomisation is the key missing feature; without it the comparison is observational, not experimental.
- 'Proving' and 'all growers': a single trial on one block cannot prove a general claim, and 'all tomato growers' is a gross overgeneralisation. The result at best supports 'drip out-yielded furrow in this block under these conditions'.
- Practical significance: 4.2 t/ha may or may not be economically meaningful; the student should compare it with the cost of converting to drip.
The honest conclusion is that the trial suggests a possible yield advantage of drip, but confounding with soil depth means the advantage cannot be attributed to irrigation, and the claim must be limited to the trial conditions.
2. The student cannot rerun the trial this season. What can be done with the existing data and how should the report be framed?
Read the worked answer
The student can still produce a useful report by treating it as a preliminary or observational comparison and being explicit about its limits.
- Measure the confounder. Record soil depth (or a proxy such as penetrometer readings or soil colour) for each plot. If soil depth differs systematically between the groups, report it and discuss it as an alternative explanation.
- Re-analyse with caution. If soil depth is measured, a simple covariate adjustment (e.g. comparing yields at similar soil depth, or a regression of yield on depth within each group) can indicate whether the irrigation difference persists. With n = 4 this is exploratory only and should be labelled as such.
- Report fully. Present all plot values, means, SDs, n and the test result, plus the confounding variable. Do not drop the marginal p-value; instead interpret it cautiously.
- Reframe claims. State the finding as 'in this block, drip plots yielded 4.2 t/ha more on average, but the groups also differed in soil depth and position, so the difference cannot be attributed to irrigation alone.' Recommend a randomised trial next season with replicated plots across the soil gradient.
Framing the work as a pilot that identifies a confounder and specifies the improved design is a legitimate and scientifically honest outcome, and is preferable to overstating a causal claim the design cannot support.