The NPS Survey That Jumped When the Channel Changed
Situation. Relationship NPS for the commercial segment came in at +44, up from +31 last quarter. Three things changed at once.
(1) The survey was embedded in the in-product support portal as well as emailed. Portal responses are 22% of the 15,600 total and average +62.
(2) The vendor changed bounce handling: hard bounces are now dropped from the invite denominator. The legacy CRM slice, about 35% of invites, had a 19% hard-bounce rate because those addresses were never cleaned.
(3) Response rate rose from 5.1% to 6.2%.
The CMO wants to publish '+13 points' in the customer update that goes out alongside the quarterly results.
1. Using the numbers given, decompose the +13 into channel mix and the remainder. What can and cannot be attributed?
Read the worked answer
Portal = 0.22 x 15,600 = 3,432 respondents at +62; email = 12,168 respondents at some value x. Solve 44 = 0.22(62) + 0.78x, so x = (44 − 13.64)/0.78 = 38.9. Email-only NPS is therefore about +39, and the channel mix contributes 44 − 38.9 ≈ 5.1 points of the 13.
The remaining ~7.9 points sits in the email channel, and that is where attribution stops. Two competing explanations cannot be separated with this data. Real improvement: a good quarter, better onboarding, faster case resolution. Selection change: the survey became easier to answer, and the marginal respondent is a different person from last quarter's respondent - plausibly someone who just opened a case and has a strong opinion, since the portal prompt is only reachable from the support experience. At a 6% response rate, 94% of invitees self-select out, so the score describes respondents, not the customer base.
Note also that portal respondents are answering immediately after a support interaction - that is transactional sentiment, not relationship sentiment, and mixing the two into one number is a construct error, not just a weighting problem.
2. Does the bounce-handling change move NPS, response rate, or both? Explain the arithmetic.
Read the worked answer
It moves the response rate only. It cannot move NPS at all.
NPS is computed on respondents. An undeliverable address produces no response whether or not it sits in the denominator, so removing it changes how the response rate is expressed and nothing about the score.
Response-rate arithmetic: dead addresses are ~35% x 19% ≈ 6.7% of all invites. If last quarter's 5.1% was computed on gross invites, the comparable deliverable-base rate was about 5.1 / (1 − 0.067) ≈ 5.5%. So this quarter's 6.2% is really a rise from ~5.5%, not from 5.1% - roughly 0.7 points, not 1.1, and part of that residual is the new portal channel bringing in easy responses.
This is a common trap in survey reporting: a denominator cleanup gets announced as engagement improvement. The defensible claim is narrow - deliverability improved and the invite base is now cleaner - and it should be stated that way. Anyone who attributes a point of NPS to a bounce-list cleanup is describing an arithmetic impossibility, which is worth saying gently and in writing before the number reaches an external audience.
3. What do you publish, and what changes next quarter?
Read the worked answer
Publish, with a footnote: relationship NPS among email respondents was +39, up from +31, on a base restated to exclude hard bounces in both quarters. Report the portal prompt separately as +62 and label it transactional touchpoint satisfaction, not relationship NPS. Do not publish +44 and do not publish +13: one is a blend of two different constructs and the other is a three-week argument waiting to happen with the first analyst who reads the methodology note.
Next quarter: pre-register the design before fieldwork - channel, invite base, bounce rule, weighting, and the exact question wording - so no choice is made after seeing results. Keep the two channels permanently separate and, if the portal prompt stays, weight it as a distinct stratum rather than letting its 22% share float with support-case volume. Add a segment weight for the legacy CRM slice so its low scores cannot be diluted by the faster-growing clean segment. Finally, add a two-field 'reason for your score' question: it is the cheapest way to distinguish a channel effect from a genuine service improvement, and it gives the CMO something to act on rather than a number to defend.