Litter programme in six schools
Six secondary schools in one Kāhui Ako agreed to compare a new zero-waste kai programme. Three schools volunteered to pilot it; the other three continued as usual. In each school, student volunteers counted visible litter items on 20 fixed 10 m transects once a week for two weeks before the programme and once a week for six weeks after. The headline result was a 38% fall in mean items per transect in pilot schools, against 6% in comparison schools. The pilot schools had higher baseline litter counts (mean 14.2 vs 8.7 items). One pilot school hosted a regional concert during the post period, and another changed its lunchtime bell. Counts were not blinded, and four post-period counts occurred on wet days. The Kāhui Ako wants to know whether it can tell parents the programme reduced litter by about a third.
1. Evaluate the claim that the programme reduced litter by about a third. Identify the strongest threats to a causal interpretation and explain how each could produce the observed pattern.
Read the worked answer
The claim is not well supported as a causal statement. Allocation was by volunteering, so the three pilot schools may differ in student behaviour, supervision, or waste infrastructure. The baseline difference matters: pilot schools started at 14.2 items per transect versus 8.7, so a larger fall can occur through regression to the mean and through having more room to improve. The comparison schools' 6% fall shows some common trend, but with only three schools per group, school-level variation is confounded with treatment. Co-events - a concert and a bell change - could affect litter independently. Unblinded student counters and wet days can bias counts downward. Treating transects within a school as independent overstates precision. A 38% versus 6% difference is suggestive, but the design cannot separate programme effect from school differences, baseline imbalance, and measurement artefacts.
2. Propose a stronger evaluation design and a matching analysis plan. Be specific about allocation, unit of analysis, and what you would report.
Read the worked answer
Use a cluster randomised trial with more schools, ideally 12–20, stratified by baseline litter and school size, and randomise within pairs. The unit of allocation is the school; the unit of analysis should reflect that, using a multilevel model with school and transect random effects, or randomisation inference. Pre-specify the primary outcome as mean items per transect at post-period, adjusted for baseline. Standardise counting: same transects, same time of day, blinded counters, rain protocol, inter-rater checks. Record co-events. Report the adjusted difference with a 95% CI and the school-level variance, not just a percentage. If randomisation is impossible, use a matched comparison and multiple pre/post waves.
3. The Kāhui Ako has limited time and cannot randomise. What quasi-experimental analysis could still support a cautious conclusion, and what would you warn against?
Read the worked answer
A comparative interrupted time series: collect several pre and post counts in all schools, fit a model with school and time effects, a post indicator, and a group-by-post term, adjusting for baseline and weather. Use transect-level counts with school random effects. Check whether the pilot and comparison schools had parallel pre-trends. Report sensitivity to excluding the concert/bell schools and wet days. Warn against quoting a precise percentage from a small, non-random sample; treating transects as independent; attributing all change to the programme; and ignoring that a higher baseline creates larger percentage falls. A cautious conclusion would say the pattern is consistent with a benefit but the design cannot establish the size.