The Report That Disagrees With Itself
An e-commerce team runs a multivariate test on a checkout page with two elements: button color (2 options) and headline (3 options), for 6 combinations. The activity has run 14 days and reports an overall conversion rate lift of +4.2% for the best combination with 95% confidence. However, the analyst notices: (1) the 'Total' row in the report shows a conversion rate that does not equal the weighted average of the combinations, (2) one combination has only 180 visitors while others have 1,200+, and (3) the revenue metric shows a large spike on day 3 that no one can explain. The team wants to ship the winning combination today.
1. Explain each of the three anomalies and what they imply for the decision to ship.
Read the worked answer
1. Total row ≠ weighted average of combinations. This usually means the report's totals include visitors or conversions that are not attributed to any experience - e.g., visitors who entered the activity but were not allocated (fallback/default content), or a metric firing outside the experience. It signals an attribution or implementation gap, so the per-combination numbers and the headline lift may not be measuring the same population.
2. One combination has 180 visitors vs 1,200+. With 6 combinations and a random allocator, traffic should be roughly even. A 7x disparity suggests either a targeting/audience rule that excludes that combination for most visitors, an implementation error where one combination's offer rarely renders, or a very recent change. Either way, that combination's result is underpowered and cannot be trusted, and the imbalance may bias the 'winner' comparison.
3. Unexplained revenue spike on day 3. A one-day spike in a revenue metric often indicates a tracking error (duplicate order events, a test order, a currency/format issue) or a genuine promotion. If it is a tracking artifact, revenue-based conclusions are invalid.
Implication: Do not ship today. The anomalies mean the experiment's integrity is in question. Investigate attribution, allocation balance, and the revenue event before trusting the +4.2%.
2. What concrete steps would you take before trusting the result, and what would you tell the team about the 95% confidence claim?
Read the worked answer
Steps:
- Reconcile the Total row against the sum of experiences; identify unallocated traffic and fix the audience/implementation gap.
- Audit allocation: check the activity's traffic allocation settings, audience rules, and whether any combination's offer fails to render (QA links per combination). Restore even allocation or restart if the imbalance is structural.
- Audit the revenue metric: check for duplicate order events, test orders, and the day-3 spike source. Exclude or correct if it is an artifact.
- Re-segment: confirm the winning combination wins within each segment and is not an artifact of unequal traffic.
On 95% confidence: Confidence is only as good as the data feeding it. A confidence level computed on contaminated totals, imbalanced allocation, and a corrupted revenue metric is not a valid basis for a decision. Confidence also says nothing about whether the result will hold in a different time period or audience. Tell the team: the number is a statistical statement about clean data; our data is not clean, so the number does not yet mean what they think it means. Re-run or repair, then re-evaluate against the planned sample and duration.