Diagnosing a stalled resolution rate at Verdant Analytics
Fin has been live at Verdant Analytics for 95 days. Resolution rate is 41% against a leadership target of 60%, and it has been flat for five weeks despite weekly content edits. Last month Fin was enabled on 3,000 conversations. A manual review of the 1,770 that Fin did not resolve produced these buckets: 34% Fin gave no answer at all, 21% Fin answered but the customer immediately asked for a human, 19% Fin gave an answer that was wrong or out of date, 14% needed an action Fin cannot take (cancel, refund, change plan), and 12% was spam or off-topic.
CSAT on Fin conversations is 4.1 versus 4.5 on human conversations. Product pricing changed three weeks ago; the canonical pricing article was updated, but a legacy comparison page and an old snippet in an internal note still exist. The team has been changing three or four things a week since launch.
1. Rank the five buckets by priority. For each of the top two, state the intervention and the rough size of the prize.
Read the worked answer
Rank by risk-adjusted value, not by size.
1. The 19% wrong answers (~336 conversations/month). Highest priority despite being mid-sized, because confident wrong answers cost trust, create rework, and - on pricing - can create commercial liability. The fix is usually cheap: find the conflicting source and remove it. Wrong answers also poison CSAT disproportionately; a customer who was misinformed rarely rates 4.
2. The 34% no-answer bucket (~602 conversations/month). Largest single pool. Split it before acting: some are genuine content gaps (write the article), some are questions that exist but are phrased unusually or are spread across two articles (improve coverage and structure), and some are unanswerable by any content (belongs in a procedure or a human). Writing 40 articles blindly will convert only part of this bucket.
3. The 14% action bucket (~248). Large structural prize but the most expensive to build - each needs a procedure, integrations, and a risk review. Worth a business case, not a sprint.
4. The 21% 'asked for a human' bucket (~372). Do not dismiss this as preference. Review a sample: in many teams it is Fin hedging, over-qualifying, or failing to state a clear answer, which is a phrasing problem, not a scope problem.
5. The 12% spam. Filter it out of the denominator or it will flatter or distort every future number you report.
2. Pricing answers are wrong even though the canonical article is current. Diagnose the likely cause and design controls that prevent recurrence, not just this instance.
Read the worked answer
Diagnosis. The canonical article was updated, but Fin retrieves from everything eligible, not from 'the article the content team thinks is authoritative'. A legacy comparison page and an internal note still contain the old numbers, and retrieval may surface them because they match the customer's phrasing more closely than the updated page does. The failure is one of source hygiene and conflicting authority, not of writing quality.
Controls, in order of strength.
- Single source of truth per fact. Archive or unpublish the legacy comparison page and remove the stale snippet from internal notes. If a retired page must remain for SEO, mark it ineligible for Fin.
- Explicit instruction. Add guidance telling Fin to answer pricing only from the canonical pricing article, to quote plan tiers exactly, and to decline and route to sales where a customer is negotiating or asking for a discount. Pricing negotiation is a sales conversation, not a support answer.
- A revenue-sensitive watchlist. Maintain a standing list of ~20 questions where a wrong answer is commercially or legally costly (pricing, refunds, data retention, SLAs, security). Review Fin's answers to these weekly, not quarterly.
- A change trigger. Any pricing, packaging, or policy change should open a checklist item that sweeps all sources, not just the main article. Add 'archive conflicting versions' as a mandatory step.
- Verify the fix with transcripts, not by re-reading the article - replay the pricing questions that failed and confirm the new answer.
3. The team changes three or four things a week. How should they test improvements so they can tell what worked and avoid regressions on topics that already resolve?
Read the worked answer
The current cadence makes learning impossible. With three or four simultaneous changes per week, the resolution rate is a lagging blend you cannot attribute. Slow down and instrument.
Build a fixed evaluation set. Take 200 held-back historical conversations, stratified across intents, and re-run them whenever something changes. Track pass/fail per intent, not an aggregate. This gives you a stable signal in hours instead of waiting a month for the live rate to move.
Change one thing at a time per topic area. Bundle edits by intent family (all pricing edits together), not by week. Record what changed, when, and which intents it targets.
Scope the measurement. Overall resolution rate moves slowly and is confounded by seasonality and traffic mix. Measure resolution rate on the targeted intents - that is where the effect appears, and it appears faster.
Protect what already works. Keep an explicit regression list of the top 30 resolving intents and re-check them after every content change. Most resolution-rate declines come from editing a good article to fix an unrelated one.
Watch the leading indicators, not just the headline. Reopen within 7 days, escalation reason, CSAT on Fin conversations, and per-step fallback rates all move before the resolution rate does. A rise in reopens means you are converting wrong answers into 'resolved' conversations, which is worse than an honest handoff.
Set a decision cadence: weekly review of the evaluation set, monthly review of live cohorts, and a written hypothesis before each change.