Checkout Conversion Drop After a Personalization Rollout
A mid-size sporting-goods retailer added a personalization layer: a 'recommended for you' product carousel on the homepage, personalized category ordering for logged-in users, and a personalized banner promoting a loyalty sign-up. Two weeks after launch, checkout conversion dropped 6% while add-to-cart stayed flat. Mobile conversion dropped 11%; desktop was roughly flat. The personalization vendor's dashboard reports 'engagement up 22%.' The team is debating whether to roll back personalization or keep it. The site has a single store view, and the personalization scripts load on every page including cart and checkout. The mobile traffic share is 68%.
1. Given flat add-to-cart but falling checkout conversion, where would you look first, and why is the personalization vendor's 'engagement up 22%' not evidence the feature is working?
Read the worked answer
Where to look first: the checkout path, not the personalization content.
The funnel shape is the clue. Add-to-cart is flat, so personalization is not suppressing discovery or product interest - people are still adding items. The loss happens after add-to-cart, which points at cart and checkout. The setup tells you the personalization scripts load on every page including cart and checkout, and that mobile is 68% of traffic with an 11% drop. That combination is the prime suspect: third-party scripts on the checkout path add weight and can interfere with checkout rendering, especially on mobile networks. The desktop-flat / mobile-down split is consistent with a payload/performance problem, not a merchandising problem.
Why 'engagement up 22%' is not evidence:
- 'Engagement' is a vendor-defined metric, almost certainly measuring interaction with the personalization widgets themselves. More clicks on a carousel is not more revenue.
- It is measured on the pages where the widgets live (homepage, category), not on checkout - the page where the business problem actually is.
- It has no control group, so it cannot separate 'personalization worked' from 'traffic mix shifted' or 'the banner is just more clickable than what it replaced.'
The vendor metric and the business metric are measuring different things, and only one of them is the goal.
First diagnostic: isolate the checkout path. Load checkout with personalization scripts disabled (or excluded from cart/checkout via a page-type rule) and compare mobile checkout completion and page-load timing against the current state. That single test separates 'personalization content is wrong' from 'personalization scripts are breaking checkout.'
2. Recommend a decision: roll back, keep, or modify - and specify what you would change and how you would prove it worked.
Read the worked answer
Decision: modify, do not roll back wholesale. The evidence points at where the scripts run, not at personalization itself. Rolling back throws away the homepage and category gains (if any) to fix a checkout problem that has a targeted fix.
Changes:
- Exclude personalization scripts from cart and checkout. Most personalization platforms support page-type exclusion. The checkout path should carry only what checkout needs. This is the highest-leverage change and is low-risk.
- Re-measure mobile checkout performance after exclusion - target the mobile checkout completion rate back toward the pre-launch baseline.
- Keep homepage carousel and category ordering, but gate the loyalty banner behind a test rather than shipping it to all users; banners are the most likely to distract at the wrong moment.
How to prove it:
- Run an A/B test on the checkout-script exclusion, split by device, with mobile checkout conversion as the primary metric and add-to-cart as a guardrail (it should not move). Two weeks minimum given the 6% effect size.
- For the loyalty banner, separate test with sign-up rate and checkout conversion as co-primary metrics - a banner that lifts sign-ups but lowers checkout conversion is a net loss and should be judged on the combined number, not sign-ups alone.
Framing for the team: the question is not 'personalization yes or no' but 'which personalization, on which pages, for which device.' The data already localized the problem to mobile checkout; the fix should be equally localized.