One Poll, Two Numbers, and a Loaded Question
A polling firm interviews 1,000 adults in one state by landline and cellphone over three weeknights and reports that 54% approve of the governor's job performance, margin of error ±3.5 points. The unweighted sample is 78% white, 61% homeowners, and 66% over age 50; turnout in the state's last midterm was 41%, and 14% among voters under 30. The firm also asks: "Do you approve of the governor's plan to cut school funding, or do you favor investing more in schools?" 62% choose the second option, and the press release says the results "show strong public opposition to the governor's education agenda." A rival firm's poll that week, using a likely-voter screen, puts the governor at 49%.
1. What can you properly infer from 54% ± 3.5 points, and what cautions apply?
Read the worked answer
- The interval: 54% with a ±3.5 margin gives roughly 50.5% to 57.5%. Approval is probably above half, but barely - the finding is consistent with a dead heat and should not be reported as a solid majority.
- Sampling error is only one error source. It assumes a probability sample and no bias. The demographic skew (78% white, 66% over 50) points to coverage and nonresponse problems that weighting can only partly correct, and weighting introduces its own variance.
- Weeknight-only fieldwork systematically under-reaches shift workers and younger adults - a plausible explanation for the age skew.
- Subgroups: 3.5 points applies to the full sample. Estimates for halves or smaller groups carry much wider intervals, so no subgroup claims should be made from this data.
- The rival poll is not a contradiction. 49% among likely voters and 54% among all adults are answers to different questions about different populations.
2. Diagnose the education question, rewrite it, and explain what a split-ballot test would show.
Read the worked answer
The question is close to unusable as written.
- Double-barreled and asymmetric: "cut school funding" versus "invest more in schools" pairs a phrase with negative connotations against one with positive connotations. No respondent would choose "cut."
- No middle option: there is no status quo, no "neither," no "don't know," and no mention of tradeoffs such as taxes or competing programs. A forced choice manufactures a majority.
- Acquiescence and social desirability push respondents toward the generous-sounding option.
A fairer version: "Some people say the state should increase school funding even if it means higher taxes. Others say funding should stay about where it is to avoid a tax increase. Which is closer to your view?" - with an explicit "neither/don't know" option.
A split-ballot test (randomly giving half the sample each version) measures how much of the 62% is wording rather than sentiment. If support falls sharply under the balanced version, the press release's claim of "strong public opposition" is a measurement artifact.
3. If the firm applied a likely-voter screen matching the state's actual midterm electorate, how would results probably move, and what new error would that introduce?
Read the worked answer
- Direction of movement: a screen built on past turnout would drop young, renter, lower-income, and less-engaged respondents. Given 14% under-30 turnout and 41% overall, the resulting electorate is older, whiter, and more homeowner-heavy than the adult sample - which would likely pull approval toward the rival firm's 49%.
- The new error: "likely voter" is a model, not a fact. Screens are built from past behavior, so they systematically miss surge elections and undercount groups with historically low but rising turnout. They can be self-fulfilling: the poll excludes the people whose behavior it is trying to predict.
- Match the population to the question. Job approval is an attitude in the adult population, so an adult sample is defensible; a vote forecast requires a likely-voter model plus explicit assumptions about turnout.
- Report both: approval among adults and vote share among likely voters are different estimates with different error profiles, and the two polls should be compared on method, not treated as a dispute over one fact.