Three Platforms, Ten Thousand Purchases, One Shopify Number
Fern & Fable is a direct-to-consumer skincare brand spending $6M a year. For Q3, Meta reported 4,800 purchases, Google Ads 3,100, and TikTok 2,200 - 10,100 in total - against 6,400 new-customer orders in Shopify. GA4 last-click shows 5,900 conversions and puts TikTok's ROAS at 1.4, the lowest of the three, while blended MER is 2.9. A brand-lift study suggests TikTok reaches 60% of new customers before their first purchase. Consent is a material factor: about 35% of EU and California traffic rejects cookies, and iOS tracking opt-in is 28%. The CFO wants to halve TikTok and move the money to Meta, whose platform-reported ROAS is 3.8. The next budget review is in three weeks.
1. Reconcile the numbers for the CFO and state clearly what should and should not be used to make the reallocation decision.
Read the worked answer
The 10,100 is not a number; it is three platforms each taking credit for the same order. Summed platform conversions routinely exceed true orders by 40–80% in a stack like this. Say that plainly: the only figures with a stable denominator are 6,400 new-customer orders in Shopify and a blended MER of 2.9.
GA4's 5,900 is not the truth either. With 35% of EU and California traffic rejecting cookies and 72% of iOS users not opting in, a large share of sessions and conversions are modeled, aggregated, or simply missed. Last-click further strips out everything before the final session - which is exactly where TikTok lives. So TikTok's 1.4 ROAS is a measurement artifact as much as a performance signal: it is being judged on a last-click window against a channel whose documented job is reaching people weeks before they convert. Meta's 3.8 is flattered in the other direction, by view-through credit and by retargeting people who were going to buy regardless.
Recommendation: do not halve TikTok on this evidence. Attribution cannot allocate budget in a consent-constrained, cross-device, four-week consideration; it can only optimize inside a channel. Allocation needs experiments and a model - a quarterly MMM calibrated against geo holdouts and platform lift tests, with MER and new-customer count as guardrails. In the interim, hold TikTok flat rather than doubling or halving it, and fund a holdout to answer the question properly. Cutting the channel with the widest new-customer reach on the weakest measurement is the reliable way to buy short-term ROAS and lose long-term growth.
2. Design the TikTok holdout. Specify the unit of randomization, the primary metric, duration, and how you will handle the power problem given 6,400 orders a quarter.
Read the worked answer
Design: a geo holdout, not per-user suppression. Select 10–14 matched DMAs or regions - matched on 12 months of new-customer orders, revenue per capita, and seasonality - and randomly assign half to keep TikTok at current weight and half to zero it for the test window. Keep every other channel identical across cells; if you also cut Meta, you learn nothing.
Primary metric: new-customer orders per capita, or new-customer revenue per capita, read as difference-in-differences against the pre-period. Secondary: branded search volume and direct traffic in the holdout, which indicate whether the channel created demand or harvested it.
Duration and power: at 6,400 orders a quarter, weekly volume is roughly 490. Split across cells, that is noisy. A test powered to detect a 10% lift would run far longer than a quarter. Be realistic - either accept a minimum detectable effect of 15–20% over 10–12 weeks and state that up front, or use a revenue-weighted metric and a longer window. Do not stop early because week three looks good; that converts a test into a story.
Read the rule in advance: if holdout geos lose materially more revenue than the media saved, TikTok is incremental and should be scaled; if they lose nothing, the channel is harvesting and should be cut. Write that decision rule before launch and pre-commit to acting on it.
One operational caution: zeroing TikTok may shift some demand into search or Meta in the holdout, which biases the result toward flat. Measure search separately so that substitution is visible rather than mistaken for proof of no effect.