The Confounded Tomato Trial
A student tests the claim that vermicompost raises tomato yield. Twenty identical pots are sown with the same variety on the same day. Ten pots receive vermicompost; ten receive none. The vermicompost pots stand on the sunlit window ledge, while the controls sit on a shaded shelf, because the ledge has room for only ten. Watering is shared between two volunteers: one waters the vermicompost pots with 250 ml daily, the other waters the controls with 250 ml on alternate days. At harvest the vermicompost pots average 14.2 fruits against 9.6 for the controls, with a standard deviation of 3.2 fruits in each group. A t-test treating the twenty pots as independent gives p = 0.004. The student writes: "Vermicompost increased yield by 48%, p < 0.05, so there is only a 0.4% chance the result is due to chance."
1. Compute the standard error of the difference and the t statistic from the figures given. State precisely what the test result establishes and what it does not.
Read the worked answer
Standard error of the difference (equal variances, n = 10 each): SE = √(s₁²/n₁ + s₂²/n₂) = √(3.2²/10 + 3.2²/10) = √(1.024 + 1.024) = √2.048 ≈ 1.43 fruits.
Observed difference = 14.2 − 9.6 = 4.6 fruits.
t = 4.6 / 1.43 ≈ 3.2, with 18 degrees of freedom. A two-tailed p of about 0.005 follows, consistent with the reported 0.004 - so the arithmetic in the report is internally sound.
What it establishes. If the only source of variation were the random assignment of pots to the two shelves, a difference of 4.6 fruits would be unlikely to arise by sampling noise alone. The difference is large relative to the within-group scatter.
What it does not establish. The null hypothesis actually tested is "no difference between these two sets of pots", which is not the same as "no effect of vermicompost". The test cannot distinguish a compost effect from a light effect, a watering effect or a watering-person effect, because all three moved with the treatment. Precision is not validity: a tightly measured confounded difference is still a confounded difference. The 48% figure is also a raw ratio (4.6/9.6 = 47.9%) on fruit count, not on fruit mass or marketable yield.
2. List the confounded variables, explain why the calculated p-value cannot rescue the design, and give a four-point redesign.
Read the worked answer
Confounded variables - each varies perfectly with the treatment, so no statistical adjustment can separate them:
- Light - sunlit ledge versus shaded shelf.
- Water volume and frequency - 250 ml daily versus 250 ml on alternate days, roughly a twofold difference.
- Watering volunteer - two people, so technique, timing and consistency differ with group.
- Position - the two groups also occupy different micro-environments (air movement, temperature), which is a consequence of 1 but is worth recording separately.
Why p cannot rescue it. A p-value answers how surprising is this difference if the groups are exchangeable, not what caused the difference. Confounding inflates or deflates the observed gap itself, so the quantity being tested is already contaminated. Increasing replication would shrink the standard error and make p smaller, not the design sounder - a confounded large experiment is worse than a confounded small one, because it lends false confidence.
Redesign.
- Randomise the twenty pots to the two treatments rather than assigning by shelf.
- Hold light constant - one bench under even light, or rotate pot positions on a fixed schedule.
- Standardise watering - same volume, same frequency, one person or a written schedule with a measuring cylinder.
- Blind the measurement and pre-specify the outcome (fruit mass, not only count); record potting mix, pot size and dates so the trial is reproducible.
A defensible target is a two-group comparison with at least 10 pots per group, randomised and identically managed, with the person harvesting fruit count and mass unaware of group assignment.
3. Rewrite the student's sentence so that it reports what the data support. Explain what is wrong with each half of the original claim.
Read the worked answer
Corrected sentence. "The vermicompost group averaged 14.2 fruits against 9.6 for the controls, a difference of 4.6 fruits (about 48%); a difference this large is unlikely under the null hypothesis of no difference between the groups (p ≈ 0.004), but because light, watering volume and the watering volunteer all differed systematically between the groups, the trial cannot attribute the difference to vermicompost."
Errors in the original.
"Vermicompost increased yield by 48%" - asserts causation from a confounded comparison. It is also a relative figure: 4.6 extra fruits on a base of 9.6. Absolute and relative framings should both be given, and yield in mass is the agronomically meaningful measure.
"p < 0.05" - reported as a threshold rather than a value; the actual value is 0.004, and threshold reporting invites the reader to think the result only just cleared the bar.
"only a 0.4% chance the result is due to chance" - the classic inversion. p is P(data this extreme | no difference), not P(no difference | data). It is not the probability that the result is due to chance, and it says nothing about the probability that vermicompost worked.
"20 pots treated as independent" - arguable if pots shared a bench and a watering can, but the deeper problem is that independence of pots does not restore independence of causes; the treatment is still entangled with light and water.