The Survey That Nobody Answered
A sixth-form student in a St Vincent town is investigating why young men aged 18 to 25 do not use a skills centre that opened 14 months ago. The centre has capacity for 120 trainees; enrolment is 31, of whom 9 are men. Her plan is a door-to-door questionnaire with 200 respondents, drawn systematically from the electoral list. In the pilot, 60 households were approached and 19 questionnaires were completed. Three men said they thought she was collecting a debt; one mentioned a recent police operation in the area. The electoral list is four years old. Her uncle manages the centre and has offered to introduce her to 'the right people'. She has six weeks, EC$800, and no letter of permission from the centre. Her teacher wants a sample that is 'representative of the community'.
1. Identify the main threats to validity in this design and state the likely direction of each bias.
Read the worked answer
1. Sampling frame. An electoral list four years old is stale in a young, mobile population; young men are less likely to be registered and more likely to have moved. The frame systematically misses the most mobile - who are also among the least likely to attend a centre.
2. Non-response (19 of 60, about 32 per cent completion). This is the most serious threat. Non-response is not random: men who distrust institutions are precisely the ones least likely to open the door, so the achieved sample is skewed towards the engaged, the employed and the trusting. Direction: barriers rooted in distrust are under-reported and the centre's appeal is over-reported.
3. Interviewer effect. A young woman with a clipboard was mistaken for a debt collector, and a recent police operation colours the area. Expect refusal and social desirability: men may under-report informal or illegal earnings and may over-report 'I'm looking for work'.
4. Timing. Door-to-door in working hours misses employed men, biasing the sample towards the unemployed and the at-home.
5. Positionality and gatekeeping. The uncle manages the centre. Respondents introduced by him will give answers designed to please him; criticism of the centre will be suppressed; and 'the right people' are his allies, not a sample.
6. Instrument. A closed questionnaire written before the pilot presupposes the reasons for non-attendance; there is no comparison group of the 89 non-attenders against the 31 enrolled.
Note what is not a valid finding: the 19 completed questionnaires cannot support a claim about the community, only about those 19 men.
2. Redesign the study within her constraints, justifying each choice.
Read the worked answer
Step 1: change the question to one she can answer. Replace 'estimate the prevalence of reasons in the community' with 'identify and test the barriers that keep young men out'. A small, purposive, mixed-methods study can answer the second; nothing in six weeks with EC$800 can answer the first. Say so in the write-up.
Step 2: multiple frames, not one. Intercept recruitment at places young men already are - bus stand, barbershops, the football field, market on Saturday, evening hours - plus a short WhatsApp link circulated through three youth groups, plus snowball from three seeds capped at three referrals each. Justification: the electoral list has already failed the pilot; intercept sampling reaches the population where it is and at the times it is there.
Step 3: shorten the instrument. An eight-question intercept survey taking two minutes raises completion far more than any incentive she can afford. Reserve depth for the qualitative strand.
Step 4: add interviews. Eight to ten semi-structured interviews with non-users, three with centre staff, two with local employers. This is where the distrust finding, which the questionnaire cannot capture, will come from.
Step 5: ethics and access. Obtain a school letter and the centre's written permission; state clearly that she is not connected to police, utilities or debt collection; guarantee anonymity; never interview anyone under 18.
Step 6: schedule. Week 1 permissions and pilot; weeks 2 to 4 fieldwork; week 5 thematic coding and simple counts; week 6 write-up and limitations.
3. Explain how she should handle her uncle's offer, and what she must report about it.
Read the worked answer
Separate the two things he is offering. Access to institutional data is legitimate and useful; introductions to 'the right people' are a gatekeeping trap.
Legitimate to accept: aggregate enrolment records, the waiting list, dropout figures, and an interview with him as a key informant, recorded with consent and reported as the manager's perspective. This is data, not a sample.
Not legitimate to accept: a recruited sample of his allies, or help selecting respondents. That would make the study a survey of the manager's network while appearing to be a survey of young men, and would remove the ability to find the thing she is looking for.
Practical handling:
- Interview young men about the centre before disclosing the connection where possible, and debrief afterwards - disclosure after the fact removes the pressure to please without deceiving anyone.
- Never let him see who said what; keep raw data with her and her teacher only.
- Where a respondent criticises the centre, have a classmate conduct that interview, or drop it and record why.
- If she concludes she cannot safely report criticism, that is a reason to change the design, not to soften the findings.
What she must report: a positionality and conflict-of-interest statement in the methodology - relationship to the manager, what access it gave, how it was managed, and what it may have affected. Also report the pilot's 32 per cent completion rate and the two men who thought she was a debt collector: those are findings about trust, not embarrassments to be hidden.