Reporting Discrepancy After a Personalization Launch
A financial-services firm launches an Auto-Target activity on its product page with three experiences and a success metric of application starts. It uses the Adobe Analytics as the reporting source (A4T). Two weeks after launch, the Target report shows Experience B winning with +9% lift and 1,200 conversions, but the Analytics report for the same period and segment shows Experience B at +2% and 950 conversions. The team also notices the Target report's conversion counts are higher than the Analytics segment totals. The activity was edited on day 8 to change Experience C's copy; the Analytics segment was built on day 1 and not updated. Traffic is steady at 200k sessions/week.
1. List the plausible causes of the discrepancy between the two reports and rank them by likelihood.
Read the worked answer
Most likely causes, ranked:
- The day-8 edit to Experience C. Editing an activity mid-flight can split or reset reporting, and the Analytics segment built on day 1 may not capture the post-edit experience mapping consistently. This is the leading suspect given the timing.
- Different attribution/segment definitions. Target's report and the Analytics segment may count conversions over different windows (e.g., session vs. visitor, or different lookback), producing different totals.
- Auto-Target allocation differences. Auto-Target shifts traffic toward the predicted-best experience over time; if the two reports snapshot allocation at different times or handle the adaptive model differently, counts diverge.
- Segment staleness. The day-1 Analytics segment may not reflect later audience or experience changes.
- Data latency / processing lag between the two systems at the time of comparison.
Ranking reflects that the edit is the only change event in the window; the others are standing conditions.
2. What steps would you take to reconcile the numbers and decide which to trust?
Read the worked answer
Reconciliation steps:
- Freeze the comparison window to a period with no edits (e.g., days 1–7) and re-pull both reports.
- Align definitions: confirm both use the same success metric, attribution model, lookback window, and audience/segment definition.
- Rebuild the Analytics segment to match the current activity (post-edit experience mapping) and re-compare.
- Check for the edit's effect on Experience C's data - if C's counts split at day 8, exclude or annotate that period.
- Compare totals (all experiences) as a sanity check; if totals differ, the issue is measurement, not the winner.
Which to trust: for A4T activities, the Analytics-sourced report is generally the system of record for conversion data, but the decision should rest on the reconciled, definition-aligned view over a clean window. If the +9% collapses to +2% after alignment, treat the result as inconclusive.
3. What governance change would prevent this on the next launch?
Read the worked answer
- No mid-flight edits to live activities; if a change is needed, pause, document, and relaunch as a new version, or accept that the test restarts.
- Build and lock the Analytics segment at launch and store its definition with the activity brief; any change requires a documented re-baseline.
- Define the reporting source of truth (A4T/Analytics) and the attribution model in the test plan before launch.
- Schedule the readout at a pre-set date with a reconciliation checklist comparing both systems before any winner is declared.