I85PREP
⌘ K
Amplitude Certification study materials
Amplitude Certification study resources

The Vanishing Checkout Events

Amplitude Certification · Put your knowledge to work in context.

All case studies

The Vanishing Checkout Events

You run product analytics for a subscription meal-kit app. Last Tuesday the team shipped a refactor of the checkout flow that moved the payment step into a new microservice. Since then, the funnel chart built on the Checkout Started and Order Placed events shows a 40% drop in orders, but the payments database shows order volume is flat week-over-week. The refactor also renamed the screen from /checkout/pay to /checkout/payment and the new service fires Order Placed with a different event_properties.flow_version. Your PM wants to roll back the release based on the funnel. The instrumentation was owned by an engineer who left last month, and no one has checked the event schema since.

1. Before agreeing to a rollback, what diagnostic steps do you take in Amplitude, and what would each result tell you?

Read the worked answer
  1. Check event volume, not just the funnel. Open the Order Placed event in the Event Segmentation chart over the last 30 days. If total volume is flat but the funnel dropped, the problem is in the funnel definition, not the product. A real regression would show a volume drop.
  2. Check the funnel's step definitions. The refactor renamed the screen to /checkout/payment. If the funnel step was defined by the old screen name or by a property tied to it, the step silently stops matching. Rebuild the funnel using event names and confirm each step still resolves.
  3. Inspect property values. Group the Order Placed event by flow_version. If the new microservice emits a new value (e.g. v2), you can confirm the new path is firing and compare conversion by version.
  4. Check for duplicate or missing events. A refactor can double-fire or drop events. Use the event stream / user timeline for a handful of test users to confirm each event fires exactly once.
  5. Validate against the source of truth. The payments DB is flat, so the product is working. That is strong evidence the instrumentation, not the product, regressed.

Conclusion: do not roll back on funnel evidence alone; fix the instrumentation first, then re-measure.

2. The team wants a guardrail so this cannot silently happen again. What governance controls do you propose, and what is the tradeoff of each?

Read the worked answer
  • Schema validation on ingest. Define a tracking plan (event names, required properties, allowed values) and reject or flag events that violate it. Tradeoff: strict rejection can drop legitimate new events; flag-and-alert is safer than hard-reject during rapid iteration.
  • Alerting on volume anomalies. Set alerts on key events (e.g. Order Placed) so a sudden drop or spike pages someone. Tradeoff: noisy alerts during low-traffic windows; tune thresholds to traffic patterns.
  • Ownership and review. Assign a data owner per event family and require a tracking-plan review in the PR that changes instrumentation. Tradeoff: process friction and slower releases.
  • Naming conventions and versioning. Standardize event and property names and keep old names as aliases during migration. Tradeoff: alias debt accumulates and must be cleaned up.
  • Reconciliation check. Periodically compare Amplitude counts to the payments DB. Tradeoff: manual effort and lag.

Recommend a tracking plan plus volume alerts as the minimum, with ownership review as the durable fix.