The Support Triage Pilot That Looked Like a Win
A 400-agent customer support organization pilots a generative AI assistant that drafts replies to inbound tickets. During an 8-week pilot on one product line, average handling time drops 22% and agent satisfaction rises. Leadership wants to roll it out to all 12 product lines next quarter. The pilot's measurement plan, however, only tracked handle time and a thumbs-up/down widget on drafts. No baseline was captured for first-contact resolution, escalation rate, or repeat-contact rate within 7 days. Agents on the pilot line were also the team's most tenured, and they self-selected into the pilot. The vendor contract auto-renews at 3x seat cost if you exceed 500 seats before renegotiation. You have 6 weeks before the next budget cycle.
1. Identify the two most decision-relevant gaps in the pilot's evidence, and explain why each could reverse the rollout recommendation.
Read the worked answer
Gap 1 - No quality/outcome baseline (FCR, escalation, repeat contact). Handle time is a process metric, not an outcome metric. A tool can cut handle time while increasing repeat contacts (customers come back because the first answer was wrong). If repeat-contact rate rose even modestly, the true cost-per-resolution could be flat or worse once you count the second contact. Without a pre-pilot baseline on those metrics, you cannot attribute any change to the tool, and you cannot detect the failure mode most likely to matter.
Gap 2 - Selection bias from tenured, self-selected agents. Tenured agents have better judgment about when to trust a draft and when to discard it, and volunteers are motivated to make the tool work. Both inflate results relative to a mandatory rollout to 400 agents across 12 lines. The 22% figure is therefore an upper bound on a best-case population, not an expected value.
Why these reverse the decision: the rollout case rests on 'it works and it will generalize.' Gap 1 attacks 'it works' (you may be trading handle time for rework). Gap 2 attacks 'it will generalize' (the effect may shrink or vanish on a representative population). Either alone can flip a go/no-go.
2. Leadership asks you to 'just roll it out and measure after.' Give a concrete counter-proposal that fits the 6-week window and the 500-seat contract threshold.
Read the worked answer
Counter-proposal: a stratified, instrumented expansion, not a full rollout.
Fix measurement first (weeks 1–2). Before expanding, pull 8–12 weeks of historical data on FCR, escalation rate, and 7-day repeat contact for all 12 lines. You can reconstruct a baseline retrospectively even though the pilot didn't capture one. This is cheap and closes Gap 1 without waiting.
Expand to a representative sample, not everyone (weeks 2–5). Pick 3–4 product lines spanning high/low complexity and mix tenured and new agents. Keep total seats below 500 to avoid triggering the 3x auto-renew - this is a hard constraint that should shape the sample size, not be discovered later.
Run a comparison, not a victory lap. Where feasible, hold out a matched group of agents on the same lines so you have a contemporaneous control. Pre-register the metrics and the decision threshold (e.g., 'ship if FCR holds within 2 points and repeat contact does not rise').
Renegotiate before scaling. Use the 6-week window to open the vendor conversation on seat pricing before you cross 500 seats, so the commercial terms don't lock you into a decision the evidence hasn't earned.
This converts a leap of faith into a decision you can defend, inside the same calendar.