GA4 and order data disagree while a PDP test sits at p = 0.09
An apparel e-commerce team redesigned its product detail page eight weeks ago. Since launch the order system shows conversion moving from 3.4% to 3.6%, while the GA4 dashboard shows 2.9%, down from 3.6% over the same period. The marketing lead has paused rollout of the redesign to the remaining 60% of traffic; the design lead argues the order data proves the change worked. In parallel the team ran an A/B test on the new PDP with 12,400 sessions per arm over three weeks: the variant converted 3.78% versus 3.41% for control, p = 0.09. Leadership wants a ship/no-ship decision this week. Average order value is $78 and contribution margin is 34%.
1. Reconcile the GA4 figure against the order data and say which number the team should be managing to.
Read the worked answer
Establish that the same thing is being measured
- Denominator: GA4 counts sessions. A PDP redesign that changed page structure, consent prompts or redirects can inflate session counts, deflating the rate without any behaviour change.
- Numerator: GA4 counts conversions only when the purchase event fires. A changed add-to-cart flow can silently break or reorder the purchase tag - orders still exist, GA4 conversions vanish.
- Consent/blocking/modelling: consent-mode modelling, ad blockers, cross-domain checkout issues and bot filtering create a persistent gap.
- Sampling: if 2.9% came from an Explorations report or a sampled view, treat it as suspect on its own.
Which is truth? Orders in the transactional system are ground truth for revenue and order count; sessions are ground truth for traffic only if instrumented correctly. Build one certified metric in the warehouse (orders ÷ sessions, single definition for both), reconcile GA4 to it weekly with a tolerance, and alert on breaks.
Diagnostic: plot purchase events against orders day by day around the launch. A step change on launch day points to tags and instrumentation; a gradual drift points to behaviour. The evidence here points to measurement breakage, in which case pausing the rollout on the GA4 number is the wrong call - but do not resume on the order number alone either until the metric is certified.
2. Interpret the A/B test result: what does p = 0.09 at 12,400 sessions per arm actually tell you?
Read the worked answer
Read the effect, then the p-value
- Control 3.41% vs variant 3.78% = +0.37pp absolute, +10.9% relative
- p = 0.09 means the 95% interval comfortably includes zero - the test has not distinguished the variant from control
Was it powered? Required n per arm ≈ 16 × p(1−p) / Δ² = 16 × 0.0341 × 0.9659 / 0.0037² ≈ 38,400 per arm. They ran 12,400, about a third of what a ~11% relative effect needs at 80% power. A non-significant result at a third of required sample is uninformative, not evidence of no effect - and stopping at p = 0.09 is a peeking decision, not a finding.
Other checks before deciding
- Sample ratio mismatch: was the traffic split actually 50/50?
- Three-week window overlapping a promotion, payday cycle or seasonal shift
- Whether the effect concentrates in one device, source or new-vs-returning segment
- Guardrails: revenue per session, add-to-cart rate, page load time, return rate - a conversion lift bought with slower pages or higher returns is not a win
Conclusion: do not ship on p = 0.09 and do not kill the change. Extend to ~40,000 per arm under a pre-registered stopping rule, and pre-commit to ship if the lower bound on revenue per session clears zero.
3. Leadership wants a decision this week. What do you recommend, and how do you frame the value at stake?
Read the worked answer
Size the prize
- A 0.37pp lift on ~100,000 monthly sessions = ~370 extra orders/month
- 370 × $78 AOV × 34% contribution ≈ $9,800/month, roughly $118k annualised
Large enough to justify a properly powered test; small enough that a false positive would not be fatal - provided guardrails hold.
Recommended path this week
- Certify the metric before any rollout decision. Fix or confirm the purchase-event instrumentation and publish a single orders-per-session figure with a fixed definition. This removes the GA4 dispute from the decision.
- Staged release with a held-out control. Roll the redesign to a further 10–20% of traffic while keeping a random 10% holdout for the full test period, so leadership sees movement and the comparison stays clean.
- Decide on revenue per session, not conversion rate alone, with returns, AOV and page speed attached as guardrails. Extend to ~40,000 sessions per arm on the holdout.
What leadership should hear: the order data suggests a real gain, the experiment is underpowered and disagrees, and the measurement layer is currently unreliable - all three resolve in four weeks with a holdout, rather than by picking whichever number is convenient this week.