The Guide That Fired Everywhere
You launched an in-app Guide (a modal survey) to collect NPS from paying users. Targeting was set to "users who viewed the billing screen in the last 7 days." On launch day, the survey response rate is high, but so is the complaint volume: users report the modal appearing on the home screen, mid-checkout, and repeatedly after dismissal. The Guide's trigger was set to fire on any screen view, and the frequency setting was left at the default. A heatmap of the billing screen shows the survey button is rarely clicked, while the modal's own dismiss button is the hottest element on the page. The PM wants to pause the Guide entirely.
1. Diagnose why the Guide is misbehaving, and propose a corrected configuration.
Read the worked answer
Diagnosis:
- Targeting vs trigger mismatch. Targeting defines who is eligible (viewed billing in 7 days); the trigger defines when it shows. The trigger was "any screen view," so eligible users see it on whatever screen they open next, including home and checkout. Targeting was correct; the trigger was not.
- Frequency default. The default frequency allowed repeated shows after dismissal, causing the repeat complaints.
- Placement. The modal is a full-screen interruption, which is why it appears mid-checkout and hurts the flow.
Corrected configuration:
- Keep the audience: paying users who viewed billing in the last 7 days.
- Change the trigger to the billing screen specifically (or a button/action on it), so it appears in context.
- Set frequency to once per user per campaign (or once per 30/90 days), and suppress after a response or dismissal.
- Exclude users in checkout or other high-stakes flows.
- Consider a less intrusive format (inline or slide-in) on the billing screen rather than a full modal.
Validate with a small holdout before full launch.
2. The heatmap shows the dismiss button is the hottest element. What does that tell you, and how does it change your read of the high response rate?
Read the worked answer
The dismiss button being the hottest element means most users are interacting with the survey only to get rid of it. That reframes the "high response rate":
- Response rate is not the same as engagement. A high response rate can be driven by an intrusive modal that forces a reaction. Check the distribution of responses and the dismissal-to-completion ratio. If dismissals dominate, the survey is an annoyance, not a signal.
- Context matters. A survey shown mid-checkout or on the home screen gets answered (or dismissed) out of context, so the NPS may reflect irritation with the interruption rather than the product.
- The heatmap is a diagnostic, not a verdict. The hot dismiss button is evidence the format and timing are wrong, not that users hate the product.
Action: pause, fix trigger and frequency, re-launch to a small holdout, and compare response quality (completion vs dismissal, response distribution) against the original. Treat the original NPS as contaminated.