The Fear-of-Crime Briefing: Ranking Wards on 214 Returns
A district administration commissions a fear-of-crime survey to justify renewing its community-policing budget. The adult population is about 240,000 across 12 wards. The team draws a sample of 600 names at random from the electoral roll and the landline directory, mails a paper questionnaire with an online link, and sends two reminders. Returns total 214 (35.7%). Officials publish a one-page briefing: “Ward 7 has the highest fear score, mean 3.9 on a 1–5 scale; Ward 7 therefore gets the new patrol unit.” The field notes record that Ward 7 is the poorest ward, has the youngest population, and supplied only 9 of the 214 returns; its non-response rate was the highest of the twelve wards. Fear was measured with a single item - “How safe do you feel walking alone after dark?” - with five options, and student volunteers doing door-to-door collection were permitted to explain items and, if asked, to fill in answers. The budget covers one further round of data collection and the renewal decision is due in ten weeks.
1. Which single threat to inference most damages the briefing’s conclusion, and how would you demonstrate the size of that damage to a non-technical administrator?
Read the worked answer
Three candidate threats, one clear winner.
- Sampling frame: the electoral roll plus landline directory systematically under-covers young, mobile and migrant adults - precisely Ward 7’s profile. This is serious but affects all wards to a degree, so it weakens the absolute estimate more than the comparison.
- Measurement: a single item plus interviewer assistance is weak but again affects every ward.
- Differential non-response: this is decisive, because the published claim is a ranking between wards, and the ward awarded the patrol unit is the one with the least data and the strongest reason to differ from those who answered. Nine cases out of 214 is 4.2% of returns for a ward that plausibly holds around 8% of adults.
Showing the damage. With 9 observations and a plausible standard deviation of about 1.1, the standard error of the Ward 7 mean is roughly 0.37, giving a 95% interval of about 3.05 to 4.75. That interval overlaps most other wards’ point estimates, so the “highest” claim is not sustained. Then do a bounded worst case: if Ward 7’s non-respondents average 2.0 instead of 3.9, the ward mean falls below 2.2 and the ranking collapses. Bias does not shrink by adding returns only from willing respondents.
The administrator should be shown the interval, not the point estimate, and told that the honest conclusion is “Ward 7 is a candidate for further measurement”, not “Ward 7 has the most fear”. Ranking twelve wards on 214 returns converts sampling noise and bias into a budget decision.
2. Design the additional round of data collection. Specify mode, sample allocation and the minimum instrument change, and justify each choice against its obvious alternative.
Read the worked answer
Sequence the problem: the frame failed before the instrument did.
- Frame and mode. Replace the electoral-roll/landline frame with an area-based frame: a random sample of addresses within each ward, from municipal tax or census enumeration blocks, with a household roster and a Kish grid to select one adult. Collect face-to-face, with a short telephone or online follow-up option for refusals. The obvious alternative - chasing the 386 non-respondents of the mail survey - has poor yield, because non-response to a mailed questionnaire predicts non-response to a reminder.
- Allocation. Use disproportionate stratified sampling: over-sample Ward 7 and the two other low-return wards so that each has at least 100 completed interviews, then weight back to population shares. Proportional allocation of the same budget would again leave Ward 7 with a handful of cases and repeat the original error.
- Instrument. Keep the old item so the two rounds can be compared, but add (a) a multi-item scale separating emotional fear, perceived risk and behavioural precaution, and (b) behaviourally anchored items (avoiding routes, locking gates, not going out). A single “walking alone after dark” item is age- and gender-loaded: it will understate fear among young men and overstate it for older respondents, which matters in a ward with the youngest population in the district.
- Administration controls. Scripted probes, no interviewer-completed answers, translation and back-translation into the local language, and a recorded response rate per ward. Report weighted estimates with intervals and stop ranking wards.
3. Interviewers were allowed to explain items and fill in answers. What specific biases does this introduce, and can you say in which direction Ward 7’s score is distorted?
Read the worked answer
The biases.
- Interviewer variance: different volunteers paraphrase the item differently, so the stimulus is no longer constant. This is a reliability failure, and its effect grows when volunteers are unevenly distributed across wards.
- Social desirability and acquiescence: a respondent being helped by a young official-sounding volunteer tends to give the answer that seems expected. Direction depends on what the volunteer signals, not on fear itself.
- Literacy accommodation: filling in answers for respondents who cannot read means the least literate households are measured by proxy, and they are concentrated in poorer wards.
- Demand effect from the setting: doorstep collection in the presence of neighbours discourages reports of fear.
Direction is not identifiable, and that is the point. One might assume assistance inflates fear (“crime is bad, isn’t it?”), but a volunteer keen to appear reassuring can as easily deflate it; a respondent who cannot read may simply agree with the first option offered. Since we cannot sign the bias, we cannot correct for it, and a comparison between wards now rests on a procedure that varied systematically with ward. That is the argument for treating the Ward 7 ranking as uninterpretable rather than merely imprecise: a confidence interval describes random error, not this.
The remedy is procedural, not statistical - fixed script, no proxy completion, and a documented refusal/conversion protocol so that the effect of assistance can at least be estimated in the next round.