Pantry Lane: a pilot that failed its primary metric but moved churn
Pantry Lane is a subscription meal-kit service with 310,000 active subscribers, $68 average monthly revenue per user, and 7.2% monthly churn. Management wants to launch a premium tier at $89.
Two pieces of evidence conflict. A survey of 2,400 subscribers found 38% 'very likely' to upgrade at that price. A two-metro pilot then ran for eight weeks with 1,100 subscribers per arm: upgrade take-up was 4.1% in control, where the tier was announced but not promoted, versus 4.6% in treatment, where it was actively promoted, p = 0.11. However, treatment-arm churn was 5.9% against 7.4% in control (p = 0.03), and revenue per subscriber rose 3.8%.
Two caveats: survey respondents were 22% high-engagement users versus 9% of the base, and the pilot metros index 130 on household income. The CFO wants a national launch in five weeks to catch the January resolution season; the CMO wants to re-test.
1. Launch, re-test, or stop? Justify the decision with the numbers, and explain why the survey and the pilot disagree so sharply.
Read the worked answer
Re-test, but not because the pilot failed - because it tested the wrong thing.
The survey-to-pilot gap is a standard intention-behavior gap amplified by selection: respondents were 2.4 times more likely to be high-engagement users than the base, and 'very likely' at a $21 premium is cheap to say. Treat 38% as a ceiling on awareness-driven interest, not a forecast; the operational estimate is nearer 4–5%.
The primary metric was underpowered to matter. A 0.5-point difference on 1,100 per arm has a standard error near 0.9 points, so the test could only have detected a difference of roughly 2.5 points. The p = 0.11 is not evidence of no effect; it is evidence of a small test.
The churn result is where the money is. Moving monthly churn from 7.2% to 5.9% lifts average subscriber life from about 13.9 to 16.9 months - roughly 21% more revenue per acquired subscriber, or about $200 at $68 ARPU. On 310,000 subscribers that dwarfs the upgrade revenue: 4.6% take-up is 14,260 upgrades at $21, about $300,000 a month gross.
But the churn effect is confounded. Control subscribers were told a tier existed and could not easily get it - a demotivating experience. The 1.5-point gap may measure resentment in control, not retention in treatment. That ambiguity, not the take-up number, is why a second test is required.
2. Design the decisive test. Specify cells, duration, primary metric, and the decision rule you would commit to before seeing results.
Read the worked answer
Design. Three cells, randomized at the subscriber level within existing metros so income skew is neutralized, 2,500 per cell for 12 weeks (long enough to see two billing cycles and the January season):
- A - true control: no tier, no mention. This is the cell the first pilot lacked.
- B - tier available, unpromoted. Measures organic uptake.
- C - tier available and promoted. Measures promoted uptake.
Primary metric: 12-week cumulative revenue per randomized subscriber, net of food and packaging cost of the premium tier. Churn and take-up are secondary and diagnostic. Committing to a revenue metric prevents the team from declaring victory on whichever secondary metric moved.
Decision rule, fixed in advance: launch nationally if C beats A by at least 4% on net revenue per subscriber with the 95% interval excluding zero, and if B's churn is no worse than A's. If only churn moves, relaunch the tier as a retention benefit bundled at no charge rather than a paid upgrade - the economics then rest on saved subscribers, not upgrade fees.
Power check: 2,500 per cell detects roughly a 1.5-point churn difference and a 3% revenue difference at 80% power. That is adequate because the decision threshold is 4%, not 0.5%.
Guardrail if the CFO insists on January: launch in the two pilot metros only, capped at 30,000 subscribers, with a pre-registered kill date at week 12 and no national media spend behind it.
3. The CFO argues that even a modest upgrade rate is free money and wants a full launch. How would you size the real upside and what would you tell her?
Read the worked answer
Size it honestly before arguing. At a realistic 5% take-up on 310,000 subscribers, that is 15,500 upgrades at $21 incremental - about $326,000 a month, or $3.9 million annualized gross. Now subtract three things the pitch omits:
- Incremental cost of goods. A premium tier carries better ingredients and packaging; if that is $9 per box, contribution falls to about $12 per upgrade, or $2.2 million a year.
- Cannibalization. Some upgraders were already high-value subscribers who would otherwise have bought add-ons. Assume 20% - call it $0.4 million.
- Support and complexity. A second SKU set raises fulfillment errors and customer-service contacts, which is where subscription businesses quietly lose margin.
Net credible contribution: roughly $1.5–1.8 million a year, not $3.9 million.
Against that, weigh the churn finding. If the tier genuinely cuts churn by even 1 point, the retention value on a 310,000 base is an order of magnitude larger than the upgrade revenue.
The message to the CFO is not 'no' but 'you are optimizing the smaller number.' Launching in January on the current evidence risks committing national inventory, packaging and media to a tier whose value may come from retention - which implies a completely different price, bundle and communication. Five weeks is enough to run the three-cell test in existing metros and still catch a February launch; it is not enough to correct a wrong positioning nationally.