The Canteen Survey That Didn't Add Up
At a secondary school in a coastal town, a lower-sixth student is completing her School-Based Assessment on whether the canteen should stay open until 4:30 p.m. She stands outside the canteen at lunch and asks the first 40 students who pass, 'Don't you agree the canteen should stay open later? It would help everybody, right?' Thirty-two say yes. She also finds a 2019 Ministry of Education report stating that after-school canteen demand was low in 14 of 18 surveyed schools. Her teacher notes that the questionnaire was never piloted and that the 40 respondents were all students who had chosen to be near the canteen at lunch. In her draft, the student writes: 'Eighty per cent of students support extended hours, so the canteen should stay open.'
1. Identify two specific threats to the validity of the student's conclusion and explain how each one operates in this scenario.
Read the worked answer
Threat 1: leading question wording. The item 'Don't you agree...?' invites a yes answer and supplies the justification ('It would help everybody'). This produces acquiescence bias: respondents may agree to be agreeable rather than because they hold the view. The result measures compliance with the question, not genuine support, so the 80% figure overstates demand.
Threat 2: biased sampling. The 40 respondents are a convenience sample of students who were near the canteen at lunch. Students who bring lunch, eat elsewhere, leave campus or cannot afford canteen food are excluded. The sample is therefore not representative of the school population, and 80% is a sample statistic, not a population parameter. The conclusion generalises from a self-selected group.
A third weakness is that the 2019 Ministry report is ignored rather than reconciled. Its finding of low after-school demand in 14 of 18 schools may reflect different conditions, but the student must explain why her school would be an exception. Validity fails when the instrument and the sample both push in the same direction: the question encourages yes, and the location selects likely supporters.
2. Explain how the student could use triangulation to strengthen her investigation, and state what triangulation would and would not prove.
Read the worked answer
She should combine methods and sources rather than rely on one lunchtime survey. For example:
- Behavioural data: canteen sales records by hour, especially 3:00–4:30 p.m., to see whether students actually buy food late.
- Interviews: short interviews with canteen staff, a sample of students who never use the canteen, and the bursar about staffing costs.
- Documentary comparison: the Ministry report, plus any school survey from the past two years.
If the survey, sales data and interviews converge, confidence in the finding increases. If they diverge, the student learns something important: for example, students may say they want extended hours but not spend money then, which changes the recommendation from 'open later' to 'pilot a late-opening day'.
Triangulation would not prove that extended hours are desirable or affordable, and it would not cancel bias if every method shares the same flaw. If she interviews only canteen users, all three methods may still overstate demand. Triangulation strengthens credibility; it does not replace representative sampling or cost analysis.
3. Recommend a revised sampling plan and one improvement to the questionnaire, justifying each choice.
Read the worked answer
Revised sampling plan. Use a stratified random sample by form level (for example, 15 students per form) drawn from the school register, and include students who bring lunch as well as canteen users. The register is the sampling frame, so every student has a known chance of selection. Stratifying by form prevents older or younger students from dominating, and including non-users tests whether demand is broad or confined to current customers. This directly addresses the lunchtime location bias.
Questionnaire improvement. Pilot the instrument with five students, then replace the leading item with neutral, behavioural and scaled questions, such as: 'How often do you buy food from the canteen after 3:00 p.m.?' (Never / Once a week / 2–3 times / Daily) and 'How much would you be willing to pay for a meal after 3:00 p.m.?' Make responses anonymous and avoid double-barrelled items.
Justification. Piloting reveals confusing wording and timing problems before the real survey. Neutral items reduce acquiescence bias. Behavioural and willingness-to-pay questions produce evidence about likely use, not just opinion. Together, the sample and instrument changes make the 80% claim defensible only if the data still support it.