Pseudoreplication in a Three-District Biochar Trial
A researcher evaluates a biochar soil amendment on paddy yield in three districts. In each district she selects one farmer field; within that field she marks six 5 m × 5 m plots, applies biochar to three chosen at random and nothing to three, and harvests a 1 m² quadrat from the centre of each plot. Pooling all districts gives 18 observations: treated mean 4.62 t/ha, control 4.11 t/ha, independent-samples t = 2.19, p = 0.032. The draft conclusion states that biochar raises paddy yield by 12% and recommends national extension. Two complications emerge in review: one control quadrat in district 2 was destroyed when a bund breached, so 17 values were used; and the district 3 field had received a subsidised fertilizer package under a separate programme, so all six of its plots yield about 25% above the other two districts. A reviewer asks for the statistical basis of the 12% figure and of the p-value.
1. Explain what the p = 0.032 actually describes and why it does not support the claim made, using the structure of the design.
Read the worked answer
What the test assumed. The t-test treats 18 quadrats (17 after the loss) as independent experimental units drawn from one population. Neither assumption holds.
What the units really are. Treatment was randomised to plots within a field, so plots are legitimate units for the within-field contrast; but plots inside one field share soil, water regime, bunds, and management, so their yields are correlated. The independent units for any claim about biochar in general are the fields - three per treatment - not the 18 quadrats. Treating quadrats as replicates is textbook pseudoreplication: it inflates the degrees of freedom from 2 to 16 and shrinks the standard error accordingly.
Sizing the effect. With a within-field correlation of even 0.2 and six plots per field, the design effect is 1 + (6 − 1)(0.2) = 2.0, so the standard error is understated by a factor of about 1.41. A t of 2.19 becomes roughly 1.55, which is not significant. The exact number depends on the unknown intraclass correlation, but the direction is not in doubt.
What p = 0.032 does say. It says that if the 17 quadrats were independent, a difference this large would arise 3% of the time under no effect. It says nothing about whether the effect generalises beyond these three fields, and it is silent on the district 3 fertilizer programme, which is confounded with any district-level comparison.
2. Set out the analysis you would actually run, state the degrees of freedom available for the treatment effect, and explain what the design can and cannot estimate.
Read the worked answer
The model. Yield ~ treatment + district + treatment:district, with field as a random intercept and the plot-level residual as the sub-plot error - a split-plot structure, because the field is the block and treatment is applied within it. The missing district 2 quadrat is handled naturally by the mixed model; no imputation is needed, and the analysis should not be run on 17 values in a balanced-design t-test while pretending nothing happened.
Degrees of freedom. The treatment main effect is tested against the field-level error, which has effectively 2 df (three fields, three districts). The treatment × district interaction also has 2 df, and with one field per district the interaction and the field-level error are the same term - they cannot be separated. Sub-plot residual df (about 9) are available only for within-field precision, not for the treatment claim.
The honest computation. Calculate the three within-field differences d1, d2, d3, take their mean and the standard error of that mean with n = 3, and use t(0.975, 2) = 4.30. The interval will be wide - quite possibly spanning zero - and that is the correct answer, not a failure.
What it cannot estimate. With one field per district, the effect cannot be separated from field-specific management, and district 3 is additionally confounded by the fertilizer programme. Report the district 3 field separately as a case where biochar was applied on top of a subsidised fertilizer regime; do not pool it into a national claim.
3. Rewrite the report conclusion so it is defensible, and specify the design change for the next season.
Read the worked answer
Replacement conclusion (58 words). 'Across three farmer fields, biochar-amended plots yielded 4.62 t/ha against 4.11 t/ha in unamended plots, a difference of 0.51 t/ha (12%). Because each district contributed a single field, field-level replication is three per treatment and the 95% confidence interval for the effect is wide; the results support a larger trial, not an extension recommendation.'
Why this wording. It keeps the observed numbers, states the unit of inference, and makes the limitation visible to a reader who will not read the methods. It also removes the phrase 'raises yield', which asserts causation the design cannot carry.
Design changes for next season
- At least four to six separate fields per district per treatment, with treatment randomised at field level, so the error term has real replication.
- Record all inputs (fertilizer, water, variety, transplanting date) per field, and either standardise them or include them as covariates.
- Pre-specify the primary outcome, the model and the minimum detectable effect before sowing; with six fields per treatment the trial can detect roughly a 10% yield difference at 80% power if the between-field CV is about 12%.
- Keep the within-field quadrats, but use them for within-field variance and spatial pattern, not as replicates.