A/B Test Results Conflict with Analytics Due to Implementation Discrepancy
A financial services company runs an A/B test on its credit card application landing page using Adobe Target. The test compares two layouts: Control (current) and Variation B (new hero image and simplified form). The primary success metric is 'application start' (clicking the 'Apply Now' button). Target reports a 5.2% lift in application starts for Variation B with 95% confidence after 2 weeks (n=12,000 per experience). However, the analytics team reports that Adobe Analytics shows only a 1.1% lift, not statistically significant. The discrepancy is traced to the fact that the 'Apply Now' button click is tracked in Target via a custom conversion event (mbox click) but in Analytics via an eVar that fires on the next page (application form). The Target activity uses a 50/50 traffic split, but the Analytics segment shows a 52/48 split. The implementation uses at.js 1.x with a global mbox and a page-specific mbox. The Analytics implementation uses Adobe Experience Platform Launch with the Adobe Target extension and the Adobe Analytics extension. The data layer pushes 'applicationStart' event only after form validation. The company wants to know which result to trust and how to align measurement.
1. Why do Target and Analytics report different lifts and traffic splits, and which metric is more reliable for this test?
Read the worked answer
The discrepancy arises from different measurement points and methods. Target's conversion event fires immediately when the 'Apply Now' button is clicked (mbox click), while Analytics' eVar fires only after the next page loads and form validation passes. This means Target counts all clicks, including those where the user abandons before the next page or fails validation, whereas Analytics counts only completed application starts. The 52/48 split in Analytics suggests either a tracking issue (e.g., some users not firing the Analytics call) or a slight imbalance due to sample ratio mismatch, but with n=12,000 per experience, a 52/48 split is within random variation (p > 0.05). The more reliable metric depends on the business goal: if the goal is to measure clicks on 'Apply Now', Target's metric is direct; if the goal is to measure actual application starts, Analytics' metric is more accurate. However, Analytics may undercount if some users don't reach the next page due to technical issues. To align, the architect should implement a single source of truth: use Analytics as the system of record for conversion, and ensure Target's success metric is mapped to the same event via the Analytics integration (A4T). Alternatively, fire the Target conversion mbox only after form validation. The 5.2% lift in Target is likely inflated by including clicks that don't lead to application starts; the 1.1% lift in Analytics is more conservative but may be diluted by tracking gaps. The architect should recommend a holdout or validation period and reconcile the data.
2. What implementation changes should the architect make to ensure consistent measurement between Target and Analytics for future tests?
Read the worked answer
- Adopt Adobe Analytics for Target (A4T) as the reporting source to ensure both systems use the same conversion event and attribution. 2) Define the success metric precisely: if 'application start' means clicking 'Apply Now', then fire the Target conversion mbox on that click and also send an Analytics event at the same time; if it means submitting the form, then fire both only after validation. 3) Ensure the Analytics eVar and Target mbox fire in the same event handler to avoid timing gaps. 4) Use the same traffic split logic: verify that the Target activity's traffic allocation is correctly implemented and that all users are bucketed before any redirects or personalization. 5) Check for flicker or race conditions: at.js 1.x with a global mbox may cause some users to see the control before the variation loads, leading to misassignment. Upgrade to at.js 2.x or Web SDK for better performance and consistency. 6) Implement a debug tool (e.g., Adobe Experience Cloud Debugger) to validate that the mbox and Analytics calls fire correctly for both experiences. 7) Set up a sample ratio mismatch (SRM) check in Analytics to detect assignment issues early. 8) Document the measurement plan and conduct a QA test before launch.