Mobile Checkout Friction Before Holiday Peak
Luma is a DTC skincare brand with 220,000 monthly sessions and 18,000 monthly orders. Mobile accounts for 71% of sessions but converts at 1.8%, versus 3.2% on desktop. Checkout has three steps: cart, shipping, payment. Funnel data shows a 46% drop from cart to shipping on mobile and a 22% drop from shipping to payment. The mobile form has 14 fields, no address autocomplete, and guest checkout is hidden below a sign-in prompt. The A/B testing tool can detect a 5% relative lift at 80% power, but the current mobile checkout test has run two weeks with no significant result. Holiday peak starts in six weeks. The growth team has one designer, one front-end developer, and limited analytics support. Paid traffic is expensive, so failed tests carry real cost.
1. Which single change should the team prioritize before holiday, and why?
Read the worked answer
Prioritize the cart-to-shipping step on mobile, specifically the hidden guest checkout and address entry friction. The 46% drop is the largest leak and sits on the highest-traffic device. A sign-in prompt placed before guest checkout can look mandatory, so tentative buyers abandon rather than create an account. Fourteen fields without address autocomplete compounds the problem on small screens.
The first release should make guest checkout the visually primary option, move sign-in to a secondary link, and add address autocomplete with country and postal-code formatting. If the team can ship only one thing, do guest checkout first because it removes a forced account decision; autocomplete is the second change.
The payment step also leaks 22%, but it is smaller and may involve payment-method trust or errors that need separate diagnosis. Do not bundle five unrelated changes into one test if the team needs to learn. However, with holiday close and low test power, shipping a low-risk friction fix as a measured release is defensible. Instrument each step and compare mobile funnel completion before and after, segmented by new versus returning users.
2. Given 18,000 monthly orders and a 5% relative minimum detectable effect, what test design is realistic before holiday?
Read the worked answer
Start with the arithmetic. Mobile is about 156,000 sessions per month. At 1.8% conversion, that is roughly 2,800 mobile orders per month. Detecting a 5% relative lift means detecting a change from 1.8% to 1.89%, or 0.09 percentage points. With standard 80% power and 95% confidence, that requires on the order of 340,000 sessions per variant. At 78,000 mobile sessions per variant per month, the test would need about four months, not six weeks.
So a 5% MDE test is not realistic before holiday. Either target a much larger relative improvement, such as 15% to 20%, or use a sequential release with pre/post measurement and qualitative evidence. A 15% relative lift on 1.8% needs roughly 38,000 sessions per variant, which is about one month at current traffic. That is still tight with seasonality.
Recommendation: ship the guest-checkout and autocomplete fixes as one reasoned release, instrument the funnel, and compare four weeks pre and post with a matched desktop or prior-year baseline. Pre-register the primary metric and avoid calling a non-significant two-week test a failure.
3. What guardrails and segments would you set, and what would make you stop the release?
Read the worked answer
Set revenue per visitor as the primary guardrail, not conversion rate alone. Add average order value, payment error rate, refund rate, support contacts per 1,000 orders, and checkout abandonment. A change that lifts completed checkouts but increases fraud, payment failures, or refunds is not a win.
Segment by mobile operating system, new versus returning customers, paid versus organic traffic, and guest versus signed-in users. New paid-traffic users are most likely to abandon at a forced sign-in, so watch that segment closely. Returning customers may react differently to guest checkout and should not be assumed to behave the same.
Stop or roll back if revenue per visitor falls materially, payment errors spike, support contacts rise, or order fraud increases. Also stop if the change causes a technical failure such as address validation rejecting valid addresses. Avoid peeking at results daily and declaring victory. With holiday traffic, monitor hourly during launch, then daily for the first week, and compare against a pre-registered baseline.