The Nineteen Per Cent Drop After the Panel Refresh
A national ratings panel has 5,200 households. Response rate has fallen from 14% to 8.4% in two years. 62% of panellists are aged 55 or over, and 16 to 34s are weighted up by a factor of 2.7; the raw 16-34 base is roughly 240 households. After a panel refresh, a youth channel's flagship show falls 19%, from a 5.8 to a 4.7 rating. Advertisers hold guaranteed deals priced on the old numbers. An audit using return-path data from 480,000 connected-TV homes suggests the show's reach is broadly stable, though that data covers one platform only and skews toward newer television sets. The ratings body offers either a larger panel of 7,000 households at 40% higher cost, or a hybrid design fusing panel data with return-path data.
1. Is the 19% fall strong evidence of a genuine decline? Show your reasoning with an approximate calculation.
Read the worked answer
Treat the rating as a proportion estimated from a small weighted subgroup. With a raw 16-34 base of about 240 households and weights averaging roughly 2.7, the effective sample is smaller than the raw count because weighting concentrates influence in few cases; a design effect around 1.2 gives an effective n of roughly 200.
For p = 0.058, the standard error is the square root of (0.058 x 0.942 / 200), about 0.0165, or 1.65 rating points. A 95% interval is therefore roughly plus or minus 3.2 points. The observed fall is 1.1 points, well inside that interval - around two-thirds of one standard error. Even taking the raw base of 240 without any design effect, the standard error is about 1.5 points and the fall remains non-significant.
Three further reasons for caution: the fall coincides exactly with a panel refresh, so a level shift caused by method change is at least as plausible as a change in behaviour; an 8.4% response rate leaves ample room for non-response bias that weighting cannot fix; and ratings are clustered within households and measured per minute, so the true uncertainty is wider than the simple binomial calculation suggests.
Conclusion: the 19% is a real number and a weak signal. It is not evidence of a 19% audience decline.
2. Advise the broadcaster on what to say to advertisers and how to present the return-path evidence.
Read the worked answer
Lead with uncertainty, not with the corroborating data. The tempting move is to say 'return-path shows reach is stable, so the panel is wrong', but that overclaims: return-path measures a different thing (reach on one platform, among connected-TV homes skewed to newer sets) and cannot be substituted for a rating without a documented fusion model. Advertisers will test that claim and it will not hold.
What the broadcaster can honestly say: the change is within the sampling error of the panel for this demographic, it coincides with a methodology refresh, and an independent reconciliation is needed before deals are repriced. Offer a concrete process - a joint audit with advertiser representation, publication of confidence intervals alongside all programme-level ratings for the next quarter, and a hold on repricing while the audit runs. Present return-path as directional corroboration only, with its coverage limitations stated in the same breath.
Commercially, the risk runs both ways. Accepting the drop triggers make-goods on a fall that may be noise; dismissing it alienates buyers who can see the number. A short interim currency with published uncertainty bands and a review clause is the least-bad option, and it protects the broadcaster's credibility when the next disputed number arrives.
3. Evaluate the choice between a larger panel and a hybrid panel-plus-return-path design.
Read the worked answer
7,000 households. Going from 5,200 to 7,000 cuts the standard error by about 16% overall, but the binding constraint is not total size: it is the 240 raw 16-34 households. Unless recruitment is targeted, a bigger panel barely moves the number that matters here, and the cost rises 40% for a marginal gain. It does preserve a single consistent methodology and full demographic coverage, which advertisers value.
Hybrid. Fusing return-path data with panel data massively increases the effective sample for platform-measured viewing and reduces dependence on extreme weights. But it introduces model dependence - the fused estimates are only as good as the imputation - plus coverage bias toward connected TVs and a currency discontinuity that guaranteed deals are not written to accommodate. The panel remains essential for who is watching, for non-connected viewing and for validating the model.
The better answer is a sequenced combination rather than a binary choice: a hybrid design with a boosted 16-34 recruitment, run in parallel with the existing panel for at least two quarters, with published reconciliation and independent audit before any currency switch. The parallel run is what makes the transition defensible; switching overnight converts a measurement dispute into a contractual one.