The 0.5% Assessment: Weighting a Survey That Missed the People Who Matter Most
A county health department serving 210,000 residents must finish its triennial community health assessment before a state grant deadline in six weeks. The 14-member coalition agreed on a single online survey, open five weeks and promoted only through the health department's social media accounts and email lists. It returned 1,100 responses - roughly 0.5% of adults. Respondents were 68% female, 71% held a bachelor's degree or higher, and median household income was $78,000 against a county median of $52,000. Only 4% live in the three ZIP codes with the county's highest pediatric asthma emergency-department rates. The health officer wants to publish the survey as the assessment's core; the coalition coordinator wants to post-stratify to county demographics and move on.
1. The coordinator proposes weighting the 1,100 responses to county age, sex, race, and income. Will that make the results usable? Explain what weighting can and cannot correct here.
Read the worked answer
No - weighting addresses one small piece of the problem.
Weighting adjusts for imbalances among people who responded. It cannot fix noncoverage: the sampling frame (department social media and email lists) gave many residents no chance to be selected - people without broadband, renters who move often, people not engaged with the department, and, notably, the asthma-burden ZIP codes.
Three failure modes to name explicitly:
- Coverage error - the frame excludes the highest-need populations; no weight recovers them.
- Self-selection - people who respond to a health survey differ from those who don't, and they differ in ways correlated with the outcome (health literacy, civic engagement, perceived need).
- Unstable cells - 4% of 1,100 is 44 respondents spread across three ZIP codes, so any ZIP-level estimate has a confidence interval so wide it is uninterpretable.
Practical rule: weighting can reduce nonresponse bias when the frame is good and nonresponse is random-ish; it cannot rescue a bad frame. Use the weighted results as one weak strand of evidence, clearly labeled, and never as the basis for allocating resources to specific neighborhoods.
2. With six weeks and no new money, what is the most defensible plan for filling the biggest gaps? Rank your moves and justify the ranking.
Read the worked answer
Rank by information gained per unit of risk, not by what is easiest to launch.
Mine existing secondary data first (week 1). State BRFSS for adult prevalence and behaviors, hospital discharge and ED surveillance for asthma, ACS for demographics and social determinants, school district health and free/reduced-price meal data, WIC and FQHC utilization, air-quality monitoring. These are already collected, already defensible, and cover the populations the survey missed. They also let you state the assessment's conclusions without relying on the online sample.
Add targeted primary collection in the three high-burden ZIP codes (weeks 2–5). Intercept surveys at the FQHC, food pantry, laundromat, and school pickup lines; two or three listening sessions held at trusted community sites with stipends, food, childcare, and interpretation. This is purposive, not representative - that is fine if you label it as such and use it to generate hypotheses and context rather than prevalence estimates.
Recruit 8–10 key-informant interviews (clinic pediatrician, school nurse, housing authority, faith leaders) for the mechanisms behind the numbers.
Write limitations before writing findings (week 6). Do not spend the six weeks chasing a representative sample you cannot obtain.
Why this order: secondary data is fast and low-risk; targeted primary data fixes the equity gap that motivated the assessment; and honest labeling protects the department from a credibility failure that would be worse than publishing a thin report.
3. The state program officer asks whether the assessment shows that the asthma disparities are 'confirmed community priorities.' What do you say, and how do you label the findings in the report?
Read the worked answer
Say: the disparity is confirmed by surveillance data; the community's ranking of it as a priority is supported by qualitative input, not by a representative survey. Those are two different claims and the report must not merge them.
Labeling conventions to use in the document:
- Indicator-level confidence tags. Mark each indicator as population-based surveillance, secondary administrative data, or non-representative community input. A reader should be able to see the provenance without reading the methods appendix.
- Ban small-cell reporting. Do not print ZIP-level estimates from fewer than ~30 respondents; aggregate or suppress.
- Name the limitation in the executive summary, not only in the methods. A one-sentence caveat in the executive summary that the online survey was a non-probability convenience sample is what prevents downstream misuse.
- Separate 'what we know' from 'what we heard.' Two-column framing.
Reasoning: funders and journalists read the executive summary and the tables. If the survey numbers are printed next to surveillance numbers without provenance flags, the assessment will be cited as if it were population-representative for the next three years, and every subsequent program decision inherits that error.