Invite Funnel Drop After Onboarding Redesign
LedgerLoop is a B2B collaboration app with 48,000 weekly active users. Six weeks ago the growth team replaced a single-step invite modal with a three-step onboarding wizard. Before the change, 62% of new workspaces saw the invite modal, 41% of those sent at least one invite, and 28% of invitees accepted within seven days. In the two weeks after launch, 71% of new workspaces saw the wizard, 38% sent an invite, and only 19% of invitees accepted. The drop is concentrated in workspaces created on mobile web. Support tickets mention invitees not receiving email, and engineering recently fixed a bug where the JavaScript SDK initialized twice on React StrictMode pages. The team wants to revert the wizard. You have event-level data, session replays, and a warehouse export.
1. Before the team reverts the wizard, what three data checks would you run, and what result in each would change the recommendation?
Read the worked answer
Start by separating numerator, denominator, and exposure. First, deduplicate events by insert_id or a composite of user, event, and client timestamp. The double SDK initialization can inflate invite_sent and workspace_created; if duplicates are concentrated after launch, the send rate and workspace denominator are distorted. Recompute after dedupe. Second, check identity resolution. If anonymous workspace creators are merged late, invite_accepted can land on a different profile or drop out of a workspace-level join. Compare raw event counts to warehouse tables. Third, check exposure and cohort maturity. The wizard may fire wizard_viewed on page load rather than actual modal render, especially on mobile web, inflating the 71% denominator. Also compare cohorts at equal age, because the latest cohort has not had a full seven days. If dedupe and identity fixes erase the gap, do not revert. If the gap survives clean workspace-level exposure and matured cohorts, the wizard is plausibly harmful.
2. The team still wants an A/B test. What randomization unit, primary metric, and guardrail would you specify, and why?
Read the worked answer
Randomize at the workspace level, not the user level. Invite acceptance is a workspace-level network outcome: one member can invite several people, and coworkers share the same onboarding path. User-level assignment would let treated and control users interact in the same workspace, contaminating the result. The primary metric should be the percentage of new workspaces with at least one accepted invite within seven days, because that is the behavior the wizard is meant to improve and it matches the decision unit. Use the pre-launch 28% as a baseline for power calculations. Guardrails should include support contacts per 1,000 new workspaces, email deliverability failures, and time to first invite. If acceptance rises but support contacts or deliverability failures also rise, the wizard is trading a metric for user pain and should not ship without a fix.