The Silent Cart Abandoners
A retailer's cart-abandonment program sends a reminder email 4 hours after abandonment. Over the last quarter it produced a 6.1% conversion rate on 48,000 monthly abandonments. The team wants to add a second reminder at 48 hours and a 10% discount at 72 hours. A junior analyst flags that 71% of all conversions attributed to the program happen within the first 6 hours, but the program's control group was disabled eight months ago when the team 'needed volume.' Leadership asks whether the 6.1% is real. Separately, the discount would cost roughly $2.10 per redeemed order against a $38 average margin, and a pilot segment shows 22% of 72-hour discount redeemers had already opened the reminder email twice without clicking.
1. Evaluate whether the 6.1% conversion rate is a trustworthy measure of the program's incremental effect, and state what you would do before expanding the program.
Read the worked answer
The 6.1% is a descriptive rate, not an incremental one. With the control group removed, the denominator is 'people who abandoned carts,' and a large share of those would have returned and purchased anyway. The 71%-within-6-hours pattern is consistent with either genuine reminder lift or with natural return behavior, and without a holdout you cannot separate them.
Before expanding:
- Reinstate a randomized holdout (e.g., 10% of eligible abandoners receive no reminder) and measure lift as treatment conversion minus holdout conversion, not raw conversion.
- Check selection effects in the pilot: discount redeemers who opened twice without clicking may be deal-seekers, not persuaded buyers.
- Set the decision metric as incremental margin per contact, not conversion rate. A 10% discount at $2.10 cost against $38 margin only pays if it moves behavior, and the 22% 'already engaged' subgroup suggests much of the discount spend is subsidizing purchases that were coming.
Do not present 6.1% to leadership as program ROI until a holdout exists.
2. Design the test structure for the 48-hour reminder and 72-hour discount so the team can isolate each element's effect.
Read the worked answer
Use a factorial or staged holdout design rather than layering both on top of the existing email:
- Cell A (control): current single 4-hour email only.
- Cell B: 4-hour email + 48-hour reminder, no discount.
- Cell C: 4-hour email + 48-hour reminder + 72-hour discount.
- Cell D (optional): 4-hour email + 72-hour discount only, to separate reminder effect from discount effect.
Randomize at the person level at the moment of abandonment, stratify by cart value and prior purchase history so the cells are comparable, and pre-register the primary metric as incremental margin per eligible abandoner with conversion and discount cost as components. Run long enough to power the smallest effect you care about - with 48,000 monthly abandonments, a few weeks gives ample volume, but the discount cell needs enough redeemers to detect a margin difference, so size on the discount-redemption rate, not overall conversion. Report per-cell lift against Cell A, and watch for fatigue/cannibalization if a customer can appear in multiple arms across months.