The $22,000 Survey the Board Wants to Publish
Lakeshore United Way commissioned a $22,000 countywide perception survey to guide a three-year strategic plan. The vendor delivered 412 interviews: 78% from an online panel, 22% from a landline sample. Residents without home internet are 19% of the county and are concentrated in the northern townships and among residents 65 and older. One key item asks: 'How much do you agree that Lakeshore United Way does important work for families like yours?' with options strongly agree, agree, somewhat agree, and disagree. The reported margin of error is ±4.8 points. Awareness in the northern townships is 31% versus 58% countywide. The board wants a goal of raising countywide awareness 20 points in three years, and the vendor's report is due to be posted publicly.
1. Identify the methodological problems and state what the data can and cannot support.
Read the worked answer
Coverage. A design that is 78% online underrepresents the 19% without home internet, and that group is concentrated exactly where the plan's attention is: the northern townships and residents 65+. The 31% versus 58% gap is more likely understated than overstated.
The margin of error is optimistic. ±4.8 points assumes simple random sampling. A mixed-mode panel has a design effect; treat ±4.8 as a floor and widen it at least 1.3 times before quoting anything.
Item construction. 'Important work for families like yours' is double-barreled - importance and personal relevance in one question. The scale also runs from 'strongly agree' to 'disagree' with no neutral point and no negative counterpart, which pushes acquiescence.
No baseline. A single wave cannot anchor a three-year trend goal.
Can support: directional regional differences, rough order-of-magnitude awareness, and a hypothesis base.
Cannot support: precise subgroup estimates, any trend claim, or a 20-point goal stated as though the baseline were exact.
2. What do you do before the report is published, and how do you handle the northern townships gap?
Read the worked answer
Before publication:
- Request the full topline, the weighting scheme, the panel source, and raw regional crosstabs from the vendor.
- Ask the vendor to reweight on age and internet access to county demographics. If they can, reissue. If they cannot, publish with explicit limitations.
- Rewrite or drop the double-barreled item so it is not a headline figure.
- Add a plain-language methods note to the public report before it goes up.
- Brief the board on limitations before they read the findings in the newspaper.
On the gap: treat 31% as a signal, not a measurement. Commission a short supplemental effort - 60–80 phone and intercept interviews in the northern townships, deliberately including landline-only and senior households - and plan from what you hear. Do not publish a township-level figure the sample cannot defend; the number will outlive the caveat, and the organization will be held to it in year three.
The general rule: fix what can be fixed before publication, disclose what cannot, and never let a flawed instrument become the plan's official baseline by default.
3. The board wants a 20-point awareness goal in three years. Advise them.
Read the worked answer
The instinct is sound; the number as written is not usable. Three problems: the baseline is one imperfect measurement with an unquantified design effect; nobody has checked whether 20 points in three years is above or below what the budget can realistically buy; and 'awareness' is undefined - unaided or aided, of the organization or of specific programs, among whom.
Recommend four changes:
- Reset the baseline with a methods-corrected survey in year 1, so the goal starts from a number the organization can defend.
- Define the metric precisely: unaided awareness of the organization and its core programs among adults 18+, measured identically each wave.
- Set the goal by segment and tie it to a theory of change. Countywide gains of 20 points imply a level of sustained reach that this budget probably does not purchase. Either scale the goal to the investment or scale the investment to the goal - and show the board that arithmetic.
- Add outcome measures - program enrollment, volunteer recruitment, donor conversion - because awareness is a means, not the result the board actually cares about.