The Support Copilot That Made Agents Slower
A software company deployed a generative-AI assistant to 600 customer-support agents. It drafts responses, summarizes tickets, and suggests knowledge-base articles. Before rollout, a vendor case study projected a 30% reduction in handle time. After four months, average handle time is up 6%, first-contact resolution is flat, and CSAT has dipped slightly. Telemetry shows agents accept the draft unchanged about 20% of the time, heavily edit it 50% of the time, and ignore it 30% of the time. Senior agents use it least. The model was evaluated on a 500-ticket benchmark against a reference answer, scoring well. Support leadership wants to expand to the sales team next quarter. The vendor contract renews in 90 days. The knowledge base has 4,000 articles, many outdated, and the assistant retrieves from it directly.
1. Diagnose the handle-time regression. What does the evaluation methodology miss, and what does the usage pattern tell you?
Read the worked answer
The benchmark measured answer quality against a reference; production measures agent workflow under time pressure. These are different constructs, and the gap explains the regression.
What the evaluation missed:
- Cost of verification. An agent cannot trust a draft without checking it. If checking takes as long as drafting, the assistant adds work. A 50% heavy-edit rate and 30% ignore rate says agents are spending time deciding whether to trust the output.
- Retrieval quality against a stale corpus. The assistant retrieves from 4,000 articles, many outdated. A confident draft grounded in a wrong article is worse than no draft: it must be detected and discarded, which is slower than starting fresh.
- Task mix. Benchmark tickets may skew toward well-documented, simple cases. Real queues include ambiguous, multi-system, and emotionally charged tickets where a draft is least useful.
- No measurement of the edit loop. The benchmark scored final answers, not the human-in-the-loop process that produces them.
What the usage pattern says:
- Senior agents using it least is the strongest signal. Experienced agents have internalized good answers and can pattern-match faster than they can evaluate a draft. The tool helps people who know least and hurts people who know most - the opposite of the intended distribution.
- The 20% accept rate is not adoption failure; it is a trust signal. Agents have correctly learned the draft is often unreliable.
The 30% vendor projection should never have been treated as a forecast; it was a marketing artifact from a different corpus, queue, and tooling.
2. Leadership wants to expand to sales next quarter. What is your recommendation, and what would you require before any expansion?
Read the worked answer
Recommendation: do not expand next quarter. Expanding a tool that is degrading the primary metric into a new function would multiply an unresolved problem and burn credibility with a second team. The renewal decision in 90 days is the forcing function; use it.
What to fix first, in order:
- Fix retrieval before touching the model. The knowledge base is the root cause of low trust. Prune or archive outdated articles, add effective-dating and ownership, and measure retrieval precision on real tickets. A better retriever over a clean corpus will move trust more than a better generator.
- Instrument the workflow, not just the output. Measure time-to-first-useful-action, edit distance, and time spent verifying, segmented by agent tenure and ticket type. The current telemetry shows behavior but not cost.
- Segment the rollout. If the tool helps new agents and hurts senior ones, target onboarding and complex-queue triage rather than universal deployment. A tool that is net-negative for experts can still be strongly positive for novices.
- Re-baseline against a control group. Four months of before/after with no control cannot separate the tool's effect from queue mix, seasonality, or staffing changes.
Gates before expansion: handle time at or below baseline for the targeted segment, accept-or-light-edit rate materially above 20%, CSAT recovered, and a documented retrieval-quality standard. Present these to the vendor at renewal - the evidence is also negotiating leverage.
Expansion to sales should be a new pilot with its own evaluation, not an extension of a support tool that has not earned it.
3. The vendor contract renews in 90 days. How should the team use the renewal, and what should change in how AI pilots are evaluated going forward?
Read the worked answer
At renewal:
- Bring the production evidence, not the case study. A 6% handle-time increase with a 20% accept rate is a concrete basis to renegotiate pricing, require retrieval-quality commitments, or reduce scope.
- Push for outcome-linked terms. Volume-based pricing rewards usage regardless of value. Seek terms tied to measurable outcomes or at minimum a pilot-exit clause with data portability, so a future decision is not hostage to the contract.
- Secure the data and evaluation rights. The company needs its ticket corpus and evaluation sets to be portable and its own, or it cannot re-evaluate alternatives.
- Consider a staged renewal (e.g., 6 months) rather than a multi-year commitment, given unresolved performance.
Changes to pilot evaluation:
- Pre-register the success metric and the baseline before rollout, including the counterfactual (control group or staggered rollout). The 30% projection should never have been the reference point.
- Evaluate in the workflow, not on a benchmark. Benchmarks are a screening tool; they do not predict productivity.
- Measure human-in-the-loop cost explicitly - verification time, edit burden, and trust calibration - not just output quality.
- Segment results by user tenure and task type so a net-zero average does not hide a harmed expert segment and a helped novice segment.
- Treat vendor case studies as hypotheses, and require the vendor to state the context (corpus, queue, tooling) so comparability can be judged.
These changes are cheap and would have caught this regression within weeks.