Abandonment and reversals in a telecom plan-change action
Vantage Mobile's postpaid support assistant uses a 14-step action called Change plan. Analytics show 38% of users abandon at step 6, which asks for the account PIN. The app channel authenticates users with SSO and passes a customer_id; the action already has a get-customer-profile extension that returns the PIN status and current plan. Step 6 is a free-text step that accepts only a four-digit PIN. When users type 'just upgrade me already' or 'the $45 one,' the action falls back to a generic 'I did not understand' message. Only 54% of started changes complete; 12% of completed changes are reversed within 24 hours, mostly because the customer chose a plan different from what they expected. The action sends one SMS and one email at the end, and there is no review screen. The business wants to reduce abandonment and reversals without removing the credit check that occurs at step 9.
1. Identify the design defects causing abandonment and reversal. For each, explain the mechanism and the change you would make.
Read the worked answer
Defects: (1) Step 6 asks for a PIN despite SSO; this is redundant and signals the assistant does not recognize the user, causing privacy concerns and abandonment. Replace it with a masked confirmation: 'I have your account PIN on file from your secure sign-in. Use it for this change?' If policy requires re-entry, explain why and offer a secure callback. (2) The PIN step uses free text with a narrow grammar; use a choice or validated input, not free text. (3) Plan selection by free text fails for 'the $45 one' because the step expects a plan name. Extract price or entity and disambiguate. (4) No review screen; 12% reversal is an expectation gap, not a payment failure. Add a confirmation summary with plan name, new monthly price, one-time charges, effective date, contract end date, and what stays the same. Require explicit 'Confirm change.' (5) Generic fallback at interruptions; map common intent phrases to the current step. (6) Credit check at step 9 after plan choice; disclose it at the start and explain possible outcomes. (7) SMS and email at the end may duplicate; confirm notification preferences. The mechanism: abandonment is effort and trust; reversal is unclear consent. Change both.
2. How should the action handle interruptions and ambiguous plan references without creating unauthorized changes?
Read the worked answer
Treat interruptions as digressions, not errors. Store the current step and slots. When the user says 'just upgrade me already,' check whether exactly one eligible upgrade exists. If yes, respond with a closed confirmation: 'I can move you to the 20GB plan at $45/month, effective today. Confirm?' If multiple plans match, show at most two eligible options with price and data, and ask a closed question. For 'the $45 one,' resolve against the customer's eligible plan set and current price, not the whole catalog; if a promo and a regular plan both show $45, disambiguate by contract term. Never treat 'just do it' as consent to a price change unless the plan and price have been stated and explicitly confirmed. Keep a boolean confirmed variable and do not call the change-plan API until it is true. If the user changes topic entirely, answer the digression, then offer to resume. If authentication is weak, do not rely on the profile; re-verify. Log the interruption phrase and resolution for tuning.
3. Design a test plan and the metrics you would use to decide whether to publish.
Read the worked answer
Test with five personas: new customer, upgrade, downgrade, family multi-line, and credit-declined. For each, script 30-40 utterances including the happy path, 'just upgrade me,' 'the $45 one,' a price ambiguity, back navigation, timeout, and a request for a human. Run in Draft on the app channel and one unauthenticated channel. Metrics: completion rate (target +15 points), abandonment at step 6 (below 10%), reversal within 24 hours (below 5%), fallback rate (below 10%), median task time, CSAT, and unauthorized-change rate (must be zero in test). Add a canary at 10% of app traffic for seven days and compare against a holdout. Stop or roll back if reversal exceeds 8%, complaints rise, or any change occurs without explicit confirmation. Review 50 canary transcripts to catch expectation gaps that metrics miss.