A Metric That Improved While Users Complained
A team ships a checkout flow and reports a 14% improvement in average order-completion time, from 3.1 minutes to 2.7 minutes, over four weeks. Simultaneously, support tickets about 'checkout froze' rose from 40 to 130 per week, and the abandonment rate went from 21% to 26%. The team's dashboard shows the improvement is statistically significant at p = 0.03. A staff engineer notes that the timing instrumentation starts when the first page loads and stops when the confirmation renders, and that a new session-timeout of 90 seconds was deployed in the same release. The team lead wants to declare success and move to the next feature.
1. Identify the most likely explanation for the divergence between the timing metric and the user-reported outcomes, and explain what the p-value does and does not tell you here.
Read the worked answer
The metrics are measuring different populations, and that is the likely cause.
Survivorship in the timing metric. Average completion time only includes sessions that completed. If the 90-second timeout now kills slow sessions, those sessions never reach the confirmation render and are excluded from the average. The remaining completions are the fast ones, so the average falls from 3.1 to 2.7 minutes - not because checkout got faster, but because slow users were removed from the sample. The abandonment rise from 21% to 26% is consistent with exactly this: roughly 5 percentage points of users are now timing out rather than completing.
What p = 0.03 says. It says that, if the underlying completion-time distribution were unchanged, you'd see a difference this large about 3% of the time. It says nothing about whether the metric is the right one, whether the sample is biased, or whether the change is desirable. A statistically significant improvement in a biased metric is still a biased metric. Significance is a statement about noise, not about validity or causation.
Confounder. The timeout shipped in the same release, so you cannot attribute the timing change to the flow redesign at all.
2. The team lead wants to ship and move on. Recommend a decision and the evidence needed to support it.
Read the worked answer
Recommendation: do not declare success. Hold the release decision, or roll back the timeout, and re-measure.
The divergence is not a rounding error - a 5-point abandonment increase on a checkout flow is a revenue event, and 130 weekly 'froze' tickets describe a real user experience that the timing dashboard is hiding.
Evidence to gather before any success claim:
- Segment the timing metric by outcome. Report completion time for all started sessions, treating timeouts as a censored outcome (e.g., report the 90th percentile and the timeout rate separately). If median completion time for all sessions rose, the 'improvement' disappears.
- Isolate the timeout. Either re-run with the timeout disabled or compare a holdout cohort. Without this, the two changes are inseparable.
- Read the tickets. Classify the 130 tickets by whether they mention freezing at or near 90 seconds. If a cluster sits at the timeout boundary, that confirms the mechanism.
- Check the abandonment funnel. Identify where users drop. If drops concentrate at the timeout boundary rather than at the redesigned steps, the redesign is not the cause of the abandonment.
Process point. The team should adopt a standing rule: any change to session handling or instrumentation requires a paired metric review, because timing metrics are uniquely vulnerable to sample-selection effects. Declaring success on a metric that improved while users complained is the failure mode the exam is testing.