Proving the Copilot Paid Off
Meridian's service desk has 1,800 agents across four countries. Eleven months ago it bought a generative AI assistant that drafts responses and retrieves knowledge articles; the licence is £2.1m a year and renewal is in eight weeks. The business case rested on a vendor pilot with 40 volunteers that claimed a 34% reduction in average handling time. Nobody recorded a control-group baseline. At month eleven, average handling time is down 9% overall but up 6% in one country; first-contact resolution is flat; customer satisfaction is flat; reuse of published knowledge articles is down 18% because agents accept the assistant's draft instead of opening the source. New-hire time-to-competence fell from 14 weeks to 11. Agent surveys show 44% say the assistant "gets in the way" on complex cases. The finance director calls it a £2.1m miss. The service director wants to expand it to field services.
1. Evaluate whether the assistant delivered value, and identify what the business case got wrong.
Read the worked answer
The 34% claim is unusable: a volunteer pilot selects enthusiasts, has no counterfactual, and compresses handling time because participants usually get an unrepresentative case mix. Without a baseline, use the strongest available design - segmented before-and-after with a comparison group drawn from queues or countries where rollout lagged - plus the one metric with a clean causal story, time-to-competence, which fell by three weeks and is hard to explain any other way.
At 1,800 agents a 9% handling-time reduction is not nothing. Convert it to money - hours saved × loaded cost - and set it against £2.1m, while being explicit that in a fixed-queue desk the benefit may realise as SLA headroom or capacity rather than cash unless headcount or volume changes. That distinction is what the finance director is actually objecting to.
What the business case got wrong: it treated a supplier pilot as evidence; it assumed handling-time reduction equates to financial value; it ignored the knowledge-base externality, where the assistant is degrading the asset it depends on; it segmented nothing, so a 6% regression in one country is hidden inside the average and points to a language or process fit problem; and it had no benefit-realisation owner or review gate before renewal, which is why the argument is happening at week 44 rather than week 4.
2. Give a renewal recommendation and the improvement actions you would attach, reasoning about conditions rather than just the price.
Read the worked answer
Renew at reduced scope, conditional - and do not expand to field services. The evidence supports one robust benefit: faster drafting and onboarding on straightforward cases. Complex cases are where 44% of agents report friction, so full-seat coverage funds the part that is not working. Negotiate a seat reduction or usage-based pricing tied to active use, and make renewal conditional on three commitments.
First, a properly designed evaluation: pre-registered metrics, a comparison group, 90 days, run by the service owner rather than the vendor. Second, a knowledge-integrity control: the assistant must cite and link the source article, and article-open rate becomes a leading indicator of knowledge decay with a threshold that triggers action. Third, a segmented quality review with agents handling complex cases, reporting by country so the regression is owned locally.
Expansion to field services should wait. Same vendor, different context, higher cost of error - an engineer dispatched on wrong information - and the current evidence base would not survive the scrutiny that a mobile field workforce attracts.
Mechanically, put this on the continual improvement register as an initiative with a hypothesis, a metric, an owner and a kill criterion, and raise the 44% sentiment and the regressing country as problem records rather than complaints. Then change procurement practice: no AI business case is approved without a pre-agreed baseline and evaluation design before go-live.