The $18,000 Flu Campaign: Is the Survey Good Enough?
Ridgeline Health, a three-hospital system serving a 240,000-person rural region, has $18,000 and six weeks to prepare a fall influenza vaccination push. The marketing director proposes reusing last year's patient experience survey: 1,900 responses, a 22% response rate, median respondent age 68, and only 9% of respondents from the two counties with the lowest vaccination rates. County health department data show adult vaccination at 38% overall - 51% in the university town, 27% in the two rural counties. The CEO wants messages tested before launch and asks whether the survey is 'good enough.' The communications team has no in-house researcher and one bilingual staff member.
1. Assess whether the existing survey can support message decisions. Name the specific validity problems and state what the data can and cannot tell you.
Read the worked answer
The survey cannot carry the message decision in the counties that matter most. The problems are specific and cumulative:
- Nonresponse bias. A 22% response rate means the 1,900 respondents differ systematically from nonrespondents - likely healthier, more engaged with the system, and more satisfied.
- Coverage error. A patient survey reaches patients. Working-age adults who rarely see a provider - a core vaccination target - are largely absent.
- Small cells. Nine percent of 1,900 is about 171 people spread across two counties, perhaps 85 each. That is roughly ±11 points at 95% confidence before any design effect.
- Construct mismatch. Satisfaction items do not measure beliefs about vaccine safety, access barriers, or trust in public health.
- Age skew. A median respondent age of 68 tells you nothing about parents of young children or adults 25–45.
What it can support: system-wide directional hypotheses and a patient-satisfaction benchmark. Decision rule: use it to generate questions, not to choose messages. If the CEO wants to proceed anyway, show the confidence intervals so the limitation is visible on the page, not buried in a caveat.
2. Build a research plan that fits $18,000 and six weeks and prioritizes the two low-vaccination counties. Show the budget logic and the tradeoffs you accept.
Read the worked answer
Sequence it so the cheapest evidence arrives first:
- Week 1 - secondary analysis ($0). County health data, prior campaign results, pharmacy flu-shot volume by ZIP, and the patient survey as hypothesis fuel only. Recruit community partners now; recruitment is the schedule risk.
- Weeks 2–3 - qualitative. Four mini-groups of 6–8 residents recruited through churches, co-ops, and the WIC office, plus six pharmacist or provider interviews. About $7,000 including facilitation, food, childcare, and modest honoraria.
- Weeks 3–5 - message test. Eight intercept sessions at grocery stores and pharmacies in the two counties, bilingual, 10-minute instrument, 200 completed interviews with $10 gift cards. About $5,500.
- Week 6 - synthesis ($0). A findings memo and two or three tested message frames.
- Contingency: $4,000 for translation, no-shows, and an extra intercept day.
Tradeoffs accepted: no probability sample, so findings are directional; no facility, so slightly less control in exchange for local credibility; small n, so report themes and verbatim patterns rather than percentages. Decision rule: where qualitative themes contradict the survey-derived hypotheses, trust the local qualitative data for message selection.
3. The CEO says: 'Just ask people in the survey whether they will get vaccinated and use that to pick the message.' Explain the problem and what you would measure instead.
Read the worked answer
Stated intent is close to useless for message selection, for three reasons. First, the intention–behavior gap is large for preventive behavior; a 'yes' in a survey overpredicts actual vaccination. Second, social desirability inflates the yes rate, especially when a health system asks. Third - and most practically - intent is not diagnostic. Learning that 60% say yes tells you nothing about which barrier to address or which message to run.
Measure the things that move behavior instead:
- Beliefs about vaccine safety and effectiveness
- Perceived personal risk and severity
- Cues to action - did a provider recommend it?
- Access barriers: cost, transportation, clinic hours, appointment friction
- Trust in the source of the recommendation
- Past vaccination behavior, which predicts future behavior better than intention does
Use an 8–10 item battery plus one open-ended barrier question. If an intent item must stay for the CEO, pair it with 'what would make that easier?' so it yields something actionable. Then test message frames against the belief items, not against intent.