Sampling, Significance and a Suspicious Survey at Halcyon Retail
Halcyon Retail wants to know whether a new store layout increases average basket value. The analyst takes the last 40 transactions from store A after the refit and the last 40 from store B, which was not refitted. Mean basket value is Tk 1,240 in A and Tk 1,150 in B, with sample standard deviations of Tk 320 and Tk 280. The analyst reports that the refit 'increased basket value by Tk 90' and recommends rolling it out to all 120 stores. Separately, the marketing team surveyed 500 customers and found that 62% preferred the new layout, and claims with 95% confidence that between 58% and 66% of all customers prefer it. The survey was conducted only in the two refitted stores, and 300 of the 500 responses came from a loyalty programme whose members receive a 10% discount. The board meets in two weeks.
1. Test whether the difference in mean basket value is statistically significant at the 5% level, and interpret the result for the board.
Read the worked answer
Step 1: state the hypotheses. Null hypothesis: the two populations have the same mean basket value. Alternative: they differ (a two-tailed test is appropriate because the analyst did not specify a direction in advance).
Step 2: standard error of the difference in means.
SE = square root of [(s1 squared / n1) + (s2 squared / n2)] = square root of [(320 squared / 40) + (280 squared / 40)] = square root of [(102,400 / 40) + (78,400 / 40)] = square root of [2,560 + 1,960] = square root of 4,520 = approximately 67.2.
Step 3: test statistic.
t = (1,240 - 1,150) / 67.2 = 90 / 67.2 = approximately 1.34.
Step 4: critical value. With roughly 76 degrees of freedom, the two-tailed 5% critical value is approximately 1.99.
Step 5: conclusion. 1.34 is well below 1.99, so we cannot reject the null hypothesis. The observed Tk 90 difference is consistent with sampling variation.
Interpretation for the board. The analyst's statement that the refit 'increased basket value by Tk 90' is not supported. A difference of this size would arise by chance reasonably often if there were no real effect. The correct statement is that the sample shows a difference of Tk 90 but the evidence is insufficient to conclude a real difference exists.
Two further cautions. First, the samples are the last 40 transactions, not random samples, so they may reflect seasonal or promotional effects. Second, the two stores differ in location, catchment and management, so any difference cannot be attributed to the layout alone. A confidence interval for the difference would run from roughly Tk 90 minus 133 to Tk 90 plus 133, that is about negative Tk 43 to positive Tk 223, which includes zero and also includes commercially trivial effects.
2. Assess the marketing team's confidence interval and its claim about customer preference.
Read the worked answer
The arithmetic is correct. For a proportion of 0.62 with n = 500, the standard error is the square root of [(0.62 x 0.38) / 500] = square root of 0.000471 = 0.0217. The 95% interval is 0.62 plus or minus 1.96 x 0.0217, giving 0.577 to 0.663, which rounds to the reported 58% to 66%.
But the interval answers the wrong question. A confidence interval quantifies sampling error only. It assumes the sample is representative of the population of interest. Here that assumption fails in two ways.
Selection bias. The survey ran only in the two refitted stores. Customers there have experienced the new layout; customers elsewhere have not. Even if every respondent answered honestly, the sample represents customers of refitted stores, not all customers of the chain. Extending the 62% to all 120 stores is an unsupported generalisation.
Incentive bias. 300 of 500 respondents are loyalty members receiving a 10% discount. Discounted customers may have different basket sizes and different preferences from full-price customers, and they may be more inclined to give answers that please the company. The effective sample is not 500 independent representative customers.
Non-response. The response rate is not stated. If 5,000 were invited and 500 replied, the 10% who chose to respond may differ systematically from those who did not.
What should be said to the board. The correct claim is narrower: among customers surveyed in the two refitted stores, 62% preferred the new layout, with a sampling margin of about plus or minus 4.3 percentage points. That is a useful signal, not a basis for a chain-wide rollout.
Remedy. Survey a random sample drawn across all stores, using a layout description or trial rather than actual experience, and report results separately for loyalty and non-loyalty customers. Until then the interval should be labelled as conditional on the sample design, not presented as a population estimate.
3. The board wants a decision in two weeks. Recommend what evidence to gather and how to frame the decision.
Read the worked answer
Separate the two questions. The basket-value question and the preference question are different, and neither is currently answered well enough to justify a chain-wide rollout costing the refit of 120 stores.
What can be said now. The basket-value difference is not statistically significant. The preference figure is a valid estimate only for the surveyed population. Neither supports rollout, and both are consistent with 'no reliable evidence of benefit yet'.
Evidence to gather in two weeks.
- Extend the transaction data. Use at least three months of transactions before and after the refit in store A, and the same period in store B, to control for seasonality. Report mean, standard deviation and sample size, and compute the confidence interval for the difference.
- Add comparison stores. Two stores is not a comparison; identify several non-refitted stores matched on catchment and size, and compare trends rather than levels.
- Control for basket composition. A higher basket value could reflect price changes, promotions or a different product mix rather than the layout. Compare units per basket and product mix.
- Fix the survey. Draw a random sample across all stores, report the response rate, and report loyalty and non-loyalty results separately.
- Cost the rollout. Set the measured uplift against the cost per store and the payback period, so the decision is framed as an investment, not a preference.
How to frame the decision. Recommend a staged rollout: refit a further small group of stores selected at random, measure against matched controls for one quarter, and roll out fully only if the uplift is both statistically significant and large enough to cover the refit cost within an agreed period. This gives the board a decision it can take in two weeks, commits limited capital, and builds the evidence the full rollout would need.
The statistical point the board should take away. A statistically significant difference is not the same as a commercially worthwhile one, and neither can be established from a sample that was not designed to answer the question being asked.