Logical Reasoning: The Faulty Correlation in a Marketing Report
A marketing analyst reports: 'Stores that ran our email campaign had 18% higher sales last quarter than stores that did not. Therefore the campaign caused an 18% sales lift; we should roll it out to all stores.' The data: 30 stores ran the campaign, 20 did not. The 30 campaign stores are in urban areas; the 20 non-campaign stores are in rural areas. Urban stores historically outsell rural stores by about 15%. The campaign cost 500,000 yen per store. The analyst's report has been forwarded to the CFO for a 25 million yen rollout decision.
1. Identify the reasoning error and explain how it affects the 18% figure.
Read the worked answer
The error is confounding: store location (urban vs rural) is correlated with both campaign participation and sales. Urban stores both ran the campaign and sell more for reasons unrelated to it.
The observed 18% gap bundles the campaign effect with the ~15% urban advantage. If the urban advantage alone explains ~15 points, the campaign's incremental effect could be as small as ~3 points - or zero, given noise. The report attributes the whole gap to the campaign, which the data do not support. This is a classic post-hoc/selection problem: the groups were not comparable before the campaign.
2. What analysis would isolate the campaign's effect, and what data would you need?
Read the worked answer
Compare like with like:
- Within urban stores, compare campaign vs non-campaign sales; within rural stores, do the same. If the campaign works, the gap should appear in both strata.
- Better: a controlled experiment - randomly assign the campaign within each location stratum and compare.
- At minimum, a before/after comparison per store (same-store sales change), not a cross-sectional comparison between different stores.
Data needed: per-store sales before and after the campaign, store location, store size, and which stores ran the campaign. Without pre-period data, even a stratified comparison is weak.
3. The CFO asks whether to approve the 25 million yen rollout now. What do you advise?
Read the worked answer
Advise not yet. The 25M rollout rests on an 18% lift that is probably mostly location effect. At 500,000 yen per store, a full rollout is a large commitment based on confounded evidence.
Recommend a small pilot: run the campaign in a random subset of stores within each location stratum for one quarter, measure same-store sales change against control stores, then decide. The pilot costs a fraction of 25M and produces evidence that actually supports a causal claim. If the pilot shows a real lift above the urban baseline, approve the rollout with confidence; if not, the 25M is saved.