The Survey That Proved Nothing
A student is investigating whether Year 12 students at her school get enough sleep. She surveys 30 students, all from her own extension English class, using a five-question online form she wrote herself. The form asks: 'Do you agree that homework stops you getting enough sleep?' with options Yes/No. She distributes it during a free period and 22 responses come back. She reports that '82% of Year 12 students are sleep-deprived because of homework' and concludes that the school should reduce homework. Her teacher notes that her class is the highest-performing group in the year and that several students were absent on the day.
1. Identify the methodological weaknesses in this research and explain how each affects the findings.
Read the worked answer
Several flaws compound. Sampling: the sample is drawn from one extension English class, which is not representative of Year 12 - it is a high-performing, self-selected group. Sample size and response: 22 of 30 is a 73% response rate, and the absent students are likely different (possibly more tired or more overloaded), producing non-response bias. Question design: 'Do you agree that homework stops you getting enough sleep?' is leading and double-barrelled - it presumes both a sleep problem and homework as the cause. Yes/No options force a false binary and give no data on actual hours slept. Construct validity: the survey never measures sleep; it measures agreement with a statement. Researcher bias: the student wrote the question and the conclusion, so the instrument is shaped toward the finding.
The critical judgement is that the reported '82%' answers a different question from the one claimed. It tells us how many respondents agreed with a leading statement; it cannot support a claim about 'Year 12 students' generally, nor establish homework as a cause.
2. Redesign the study to produce more credible findings, justifying each change.
Read the worked answer
Sampling: draw from the whole Year 12 cohort using a random or stratified method (e.g., select students across all classes and achievement levels), aiming for a larger sample and recording who does not respond so non-response can be reported. Instrument: replace the leading question with neutral, measurable items - 'On a typical school night, how many hours do you sleep?' (numeric bands), 'How many hours of homework do you do on a typical weeknight?', and a Likert item on perceived sufficiency, with balanced response options. Piloting: trial the survey on a small group outside the sample to catch ambiguous wording. Timing and mode: distribute online over a week so absent students can respond, and assure anonymity to reduce socially desirable answers. Analysis: report sleep hours and homework hours separately and describe the relationship cautiously, acknowledging that correlation does not establish causation.
Justification: each change targets a specific threat - representativeness, bias, validity, and overclaiming. The redesign will not prove causation either, but it will support a defensible descriptive claim, which is the appropriate limit for a school-based survey.