The Shelf Test That Almost Passed
A grocery chain tested a new shelf layout in 8 stores, matched with 8 control stores on baseline sales, format and catchment. The trial ran 8 weeks. Baseline sales averaged ₹9.2 lakh per store per week, and gross margin is 22%. Comparing each test store's change against its matched control, the average lift was +2.4 percentage points, with a standard deviation of the store-level differences of 3.1 percentage points. The trial cost ₹6.4 lakh. Rolling the layout out across 210 stores would cost ₹1.6 crore in fixtures and planogram resets, and the layout can be reversed. The marketing head wants a decision at the next meeting: she is being pressed by the CFO, who points out that a 2.4% lift with only 8 stores 'is not statistically significant, so we learned nothing'. The category manager adds that two of the eight test stores showed a decline of over 3%.
1. Test the result formally at the 5% level. What is the test statistic, the degrees of freedom, and the correct interpretation of a non-significant result here?
Read the worked answer
Set-up. Paired design: one difference per test store (test change minus matched control change), n = 8. Hypotheses: H₀: mean lift = 0; H₁: mean lift ≠ 0. Two-tailed, α = 0.05.
Test statistic. SE = s / √n = 3.1 / √8 = 3.1 / 2.828 = 1.096. t = 2.4 / 1.096 = 2.19, with df = 7.
Critical value. t₀.₀₂₅,₇ = 2.365. Since 2.19 < 2.365, fail to reject H₀. The p-value is roughly 0.065 - significant at 10%, not at 5%.
The interpretation is where most answers go wrong. Failing to reject is not evidence that the layout does nothing. With 8 stores the study is underpowered: the standard error is wide enough to be consistent with a true effect anywhere from roughly −0.2 to +5.0 percentage points (95% interval: 2.4 ± 2.365 × 1.096). The data are compatible with both 'no effect' and 'a very profitable effect'.
The CFO's error is treating a decision threshold as a verdict on truth. 'Not significant' answers a question nobody asked - 'could this plausibly be zero?' - when the decision question is 'what should we do given what we now believe?'
Also flag the design issues: 8 weeks may capture novelty effect and does not cover a full seasonal or promotional cycle; the two stores with declines over 3% need checking for a confound (a local competitor opening, a stock-out, a planogram compliance failure), because if they are data errors the lift and the standard deviation both change materially.
2. Setting statistics aside, what is the right commercial decision? Quantify the expected value of rollout, the cost of being wrong, and the value of more information.
Read the worked answer
Value if the effect is real. Weekly sales per store = ₹9.2 lakh; annual = ₹478.4 lakh. A 2.4% lift = ₹11.48 lakh of extra sales per store per year. At 22% gross margin = ₹2.53 lakh contribution per store per year. Across 210 stores = ₹5.30 crore a year.
Cost of rollout. ₹1.6 crore one-time → payback about 3.6 months, and the investment is largely reversible (fixtures can be moved; planograms can be reset).
Cost of being wrong. If the true effect is zero, the loss is the ₹1.6 crore rollout plus disruption - but a meaningful part of the fixtures retains value, so the true downside is well under ₹1.6 crore. Note the asymmetry: the upside is ₹5.3 crore recurring, the downside is a partly recoverable one-off ₹1.6 crore. Even at a 40% probability that the effect is real, expected value is strongly positive (0.4 × 5.3 = ₹2.12 crore against a downside under ₹1.6 crore).
Value of more information. To detect 2.4 points with s = 3.1 at 80% power and α = 0.05 two-tailed: n per arm = 2 × (1.96 + 0.84)² × 3.1² / 2.4² = 2 × 7.84 × 9.61 / 5.76 ≈ 27 stores per arm. Extending the trial is cheap relative to ₹1.6 crore.
Recommendation: a staged rollout - 40 stores in the next quarter, chosen to include at least one store resembling the two decliners - with a pre-committed review date. This buys the sample size the statistician wants while capturing most of the value now, instead of spending three months proving a point worth ₹5 crore a year.
3. The CFO proposes re-running the trial in 60 stores for 6 months to obtain a definitive answer. Evaluate this proposal, including what you would change in the design.
Read the worked answer
The proposal is statistically sound and commercially expensive. Sixty stores per arm comfortably exceeds the 27 needed, so power is not the issue. The issue is timing: six months of delay defers roughly ₹2.65 crore of contribution (half a year of ₹5.3 crore) to protect against a downside already shown to be modest and partly recoverable. That is not a good trade unless the rollout is far less reversible than described.
Design changes that matter more than sample size:
- Length, not just breadth. Run a full 12 months or at least two quarters, so seasonality and the post-Christmas reset are captured, and so novelty effect can be separated from a durable shift. Eight weeks cannot distinguish the two.
- Pre-register the primary metric. Fix one outcome (sales per store per week, or category gross profit) before the trial. Testing several metrics after the fact guarantees a false positive somewhere.
- Check compliance. In layout trials the most common failure is that stores drift back to the old planogram. Audit compliance weekly; without it you are measuring intentions, not layout.
- Stratify by the two decliners' characteristics and by store size, so the analysis can say where the layout works, which is more useful than a single average.
- Randomise, don't match. Random assignment of stores to test and control removes the risk that matching on baseline sales selected stores already trending differently.
- Model the decay. Fit the lift week by week. If it is falling from +4% in week 2 to +2% in week 7, the long-run effect is smaller than the headline and the rollout case changes.
Better decision rule: ask the CFO to approve the staged 40-store rollout and, in parallel, keep 30 of those stores instrumented as a long-run measurement group. That converts the delay into evidence at no incremental cost.