The Pilot That Looked Like Proof
A retail chain tested a new checkout layout in 12 of its 340 stores over eight weeks. Sales per transaction rose 6.2% in the pilot stores versus 1.1% chain-wide. The pilot stores were chosen by regional managers who volunteered them, and they are disproportionately in affluent suburbs. The test ran during a promotional period that included a chain-wide discount event in weeks five and six. Store managers knew they were being measured and received a visit from the COO midway through. The analytics team reports the result as 'statistically significant' with a p-value of 0.03. The COO wants to roll out to all 340 stores next quarter at a cost of $1,400 per store, citing the 6.2% figure. The CFO asks whether the result would hold in the 200 stores that are in lower-income areas.
1. Evaluate the strength of the pilot evidence and identify the specific threats to its validity.
Read the worked answer
The headline 6.2% is not yet a credible estimate of chain-wide effect. Selection bias: stores volunteered by regional managers are not a random sample, and the affluent-suburb skew means the pilot group differs from the 200 lower-income stores on exactly the variable the CFO is worried about. Confounding: the chain-wide discount event in weeks five and six affects pilot and comparison stores, but if pilot stores are in higher-income areas their response to a discount may differ, so the comparison is not clean. Hawthorne effect: managers knew they were measured and received a COO visit, which can independently lift performance. Small sample and short window: 12 stores over eight weeks gives limited power and no read on durability. The p-value of 0.03 addresses only sampling noise; it says nothing about these biases, and with many metrics examined it may also reflect multiple comparisons. The honest reading: the pilot is suggestive, not decisive, and the 6.2% should be treated as an upper bound rather than a forecast.
2. Design the cheapest next step that would let the chain decide about the lower-income stores, and state what result would change the rollout decision.
Read the worked answer
The cheapest informative step is a stratified, randomized extension: randomly assign the new layout within matched pairs of stores - pairing on income profile, store size, and baseline sales - and run for at least eight weeks outside any chain-wide promotion. If a full extension is too costly, a matched-pair comparison in 20 lower-income stores against 20 matched controls costs roughly $28,000 in fixtures plus measurement, far below the $476,000 full rollout. Pre-register the metric (sales per transaction) and the analysis before launch. The result that should change the decision: if the effect in lower-income stores is materially smaller - say below 2% - the rollout's payback likely fails at $1,400 per store and should be limited to the segments where the effect replicates. A second decision rule: if the effect in the randomized extension falls below the 1.1% chain-wide baseline, the original pilot result should be treated as an artifact and the rollout halted. Stating these thresholds in advance prevents the team from rationalizing whatever comes back.