The Pilot That Wasn't Quite a Trial
You are reviewing a 380-word evidence brief that a training provider has sent to a secondary school. The brief claims that a 12-week pilot 'proves' weekly 20-minute vocabulary quizzes raise exam pass rates by 18 percentage points. The pilot involved 48 volunteer learners who chose to take part; the end-of-course survey had a 31% response rate. There was no control group. The pilot cohort's pass rate was 64%, compared with 46% for the previous year's cohort. The provider names two partner schools but gives no dropout figures and no attendance data. The word 'significant' appears three times, but no statistical test is reported. The brief was published in 2023; the data were collected in 2021, when remote learning was widespread. Your school has 600 learners and is considering adoption. The head teacher wants a decision within a week and has asked you to advise on whether the evidence is strong enough and what to request before any pilot.
1. Identify the two weaknesses that most seriously limit the claim that the quizzes caused the pass-rate rise. Explain how each weakness affects the inference.
Read the worked answer
The two most serious weaknesses are selection and confounding. The 48 learners volunteered, so they may already have been more motivated or more able than the 600 learners the school would target. That makes the 18-point gap an unreliable estimate of what the scheme would do elsewhere. Second, the comparison is historical: the 46% baseline comes from the previous year, when remote learning was widespread, while the 64% figure comes from 2021 pilot conditions. Any change in teaching, assessment difficulty, or cohort composition could produce the gap without the quizzes causing it. The 31% survey response rate also raises non-response bias, and the missing dropout and attendance data mean we cannot tell whether the scheme was delivered as described. The word 'significant' is rhetorical here: no test is reported, so it should not be read as statistical significance.
2. Write a recommendation of about 60 words to the head teacher. State whether to adopt, pilot, or reject, and name two pieces of evidence to request.
Read the worked answer
Recommendation (68 words): Do not adopt the scheme across 600 learners on this evidence. Fund a small, time-limited pilot with a control or matched comparison group, and require the provider to supply attendance, dropout, and full response-rate data. Ask for the raw pre- and post-test scores and the statistical test behind 'significant'. Reassess after one term. This protects the school from scaling an unproven intervention while keeping the option open.
Why this wording: it separates the decision (not adopt now) from the evidence request (what would change my mind), and it names concrete data rather than a general demand for 'more research'.