The Substituted Quadrat: An Investigation That Overclaims
A student investigates: 'Does a formal walking trail reduce trampling damage in a conservation park?' Method: five sites spaced along a 2 km trail; at each site, three 1 m² quadrats (on-trail, 1 m from the edge, 5 m from the edge); measured soil compaction by penetrometer drop, vegetation cover percentage, and litter cover. Two sampling days were lost to rain. At site 4, a fallen branch blocked access to the 5 m quadrat, so the student substituted a quadrat near the car park. Results: compaction was highest on-trail at all five sites; vegetation cover at 1 m was lower than at 5 m at only two of five sites. The draft conclusion reads: 'The trail causes damage to vegetation.'
1. Classify the weaknesses as threats to validity or to reliability, and explain how each one limits what the student can claim.
Read the worked answer
Validity threats (the study is not measuring what it claims)
- No measure of use. The assumed cause is trail presence, but visitor numbers almost certainly vary along 2 km and between sites. Without a use measure, trail presence is confounded with intensity of impact.
- The substitution confounds the comparison. A car-park quadrat differs in soil, light, compaction history, and use. Site 4 is no longer a like-for-like pair, so the paired contrast is broken at one of only five sites.
- Moisture confounds the penetrometer. Penetrometer resistance depends strongly on soil water content, and sampling followed rain. Differences in compaction may partly reflect how wet each site was on the day.
- Single season, single park. The claim generalises far beyond the data.
Reliability threats (the measurement is not stable or repeatable)
- No replication within sites. One quadrat per position per site means a single unusual patch swings the result; n = 5 is effectively the sample size for the trend.
- Observer estimation of cover by one person with no calibration or duplicate scoring.
- Inconsistent conditions from lost rain days, so quadrats were not measured under comparable circumstances.
Consequence: the compaction finding is the most robust element because it is consistent across all five sites and uses an instrument. The vegetation finding is not consistent (two of five), so the draft conclusion is unsupported by the student's own data.
2. Rewrite the conclusion so it is defensible on this data, and name the single methodological change that would most increase confidence in it.
Read the worked answer
Defensible conclusion 'At the five sites sampled, soil compaction was consistently higher on the trail than at 1 m and 5 m from the edge, which is consistent with trampling pressure. Effects on vegetation cover were inconsistent across sites, so this study does not establish a vegetation effect. The results are limited to one trail in one season and do not account for variation in visitor use.'
Why this version works: it separates what the data show (a consistent compaction gradient) from what they do not (a vegetation effect), states the direction of the evidence honestly, and names the boundary. A conclusion that reports a null or mixed result accurately scores better than one that forces a pattern.
Single highest-value change: replication. Replace the single quadrat per position with three to five replicate quadrats per position per site, and increase the number of sites.
Reasoning for that choice: the largest weakness is not instrument precision - it is that a real effect cannot be distinguished from site-to-site noise, particularly for vegetation cover, which is patchy. Replication attacks that directly. Measuring visitor use would improve causal interpretation, but even a perfect use measure with one quadrat per site would still leave the result vulnerable to one anomalous patch.
3. Should the substituted car-park site be excluded, retained but flagged, or discarded and re-sampled? Justify your choice.
Read the worked answer
Recommendation: retain it, flag it explicitly, and run the analysis both with and without it.
Reasoning
- Excluding it silently is the worst option. It hides a decision that was made in the field and that influenced the data. A reader cannot then judge whether the conclusion depends on that choice, and the investigation looks cleaner than it was.
- Discarding and re-sampling is better but not automatically best. It costs time, and the replacement quadrat would still be a 5 m-from-edge position chosen after the original was blocked - the selection is still non-random. It is worth doing only if there is time and the original site can genuinely be accessed.
- Retaining with a flag is defensible because it is transparent, and because a sensitivity check answers the real question: does the conclusion survive the questionable point?
Method for the flag
- Note in the method the date, the obstruction, and the reason for substitution.
- Mark site 4 as confounded in the results table.
- Report the trend with site 4 included and excluded.
- If the compaction gradient holds both ways, state that the finding is robust to the substitution. If it does not, say so and treat site 4 as a limitation.
Better option still: retain the car-park quadrat as a separate 'high-use comparison' rather than folding it into the trail-edge comparison. It then adds information about intensive use instead of corrupting a paired design.