The Loyalty Survey That Overstated Satisfaction
A regional gym chain with 12,000 members emails a satisfaction survey to all members; 1,840 responses arrive. The marketing manager reports '92% of members are satisfied' and plans to raise prices. You notice the survey was emailed only to members who opted into marketing emails (about 6,400 people), that the response link was also posted in a members-only Facebook group, and that 38 of the 1,840 responses came from a single IP address within nine minutes. The member database also shows 214 duplicate records created when the same person signed up online and in-branch. The manager says the numbers are 'close enough' and the deadline is Friday.
1. Identify the data integrity and collection problems in this scenario and explain how each could bias the 92% figure.
Read the worked answer
Four distinct problems, each with a different bias mechanism:
Coverage/selection bias. The survey reached only the 6,400 marketing opt-ins, not all 12,000 members. Members who opt out of marketing may differ systematically (e.g., less engaged, or privacy-conscious). The 92% describes opt-ins, not the membership, so generalising to all 12,000 is unsupported.
Self-selection bias. Only 1,840 of ~6,400 responded (~29%). People who respond to satisfaction surveys are not a random sample; both very happy and very unhappy members are over-represented. The Facebook group posting adds a non-probability channel, so the denominator is now unclear.
Duplicate/fraudulent responses. 38 responses from one IP in nine minutes is implausible as independent members and should be quarantined or investigated. The 214 duplicate member records also mean any per-member rate is computed against an inflated or deflated denominator, so percentages drift.
Accuracy vs. validity. Even if each response is accurately recorded (accuracy), the instrument may not measure what is claimed (validity) - 'satisfied' on a single item is not the same construct as willingness to accept a price rise.
The 92% is a real number describing a self-selected subset; it is not an estimate of member satisfaction. The correct response is to report the figure with its scope and limitations, or re-collect with a probability sample.
2. The manager wants to proceed. Outline a defensible alternative you could deliver before Friday, naming specific validation rules and cleaning steps.
Read the worked answer
A defensible deliverable is possible in the time available if you change the claim, not the data.
Cleaning and validation before any analysis:
- De-duplicate the member table on a composite key (surname + date of birth + mobile), flagging the 214 suspected duplicates for manual review rather than blind deletion.
- Apply a range check on satisfaction items (1–5 only), a consistency check (a respondent cannot be 'very satisfied' and 'likely to cancel'), and a completeness check on key fields.
- Quarantine the 38 same-IP responses into an excluded set and report them transparently; do not silently delete them.
- Recompute the denominator as 'members invited and eligible', and report the response rate (~29%).
Reporting changes:
- Restate the headline as '92% of the 1,840 respondents reported satisfaction', with the response rate, channel mix, and exclusions shown.
- Present a confidence interval or at minimum a sensitivity note: if non-respondents were less satisfied, the true figure could be materially lower.
- Separate the price-rise decision from the survey; recommend a short random-sample pulse survey (e.g., 400 randomly selected members) as the evidence base for pricing.
This is honest, fast, and protects the organisation from a decision built on a biased sample.