The Survey That Proved the Wrong Thing
A retail chain runs a customer satisfaction survey after every purchase. Over one quarter it receives 4,200 responses out of 61,000 transactions. Management notices satisfaction scores are highest on weekends and concludes that weekend staffing creates a better experience, and proposes cutting weekday staff to fund more weekend coverage. A junior analyst is asked to sanity-check the claim. The survey is optional, sent by email, and customers who had a complaint are known anecdotally to be more likely to respond. Weekday transactions are 70% of the total and skew toward older, less email-active customers.
1. Identify the threats to the validity of the weekend conclusion.
Read the worked answer
Three threats stack up. First, non-response bias: only 6.9% respond, and if complainers respond more, the sample is not representative of all customers. Second, confounding: weekend and weekday customers may differ (leisure vs. routine shoppers, age, email habits), so any satisfaction gap could reflect who shops, not the staffing. Third, the response rate itself may vary by day - if weekday response is lower, the weekday sample is even less representative. Concluding 'weekend staffing causes higher satisfaction' from observational survey data confounds correlation with causation.
2. Design a cheap check the analyst could run before any staffing change.
Read the worked answer
Compare response rates by day of week and compare the demographic/transaction profile of responders vs. non-responders using transaction data already held. If weekday response is much lower or responders skew young/email-active, the comparison is unreliable. Also check whether the satisfaction gap survives controlling for customer type (e.g., first-time vs. repeat, basket size). If the gap disappears once these are held constant, the staffing hypothesis loses support. None of this requires new data collection - it uses existing transaction records.
3. What would a genuinely causal test look like, and why recommend it before cutting staff?
Read the worked answer
A controlled pilot: pick comparable stores, add weekend staff in some and hold others constant, and measure satisfaction and sales before/after. This isolates the staffing effect from who-shops-when. It matters because the proposed action - cutting weekday staff - has real cost and risk (longer weekday queues, lost sales) based on evidence that may be pure selection bias. The decision rule should be: act only when the effect survives a design that rules out the confound.