Pilot scope under a six-week deadline at Northwind Labs
Northwind Labs is a B2B SaaS company handling 6,200 conversations a month across chat, email, and a legacy web form. Eleven agents cover 24/5. The help center holds 900 articles, last audited 20 months ago. A 300-conversation sample found that 46% of volume is four topics: password and MFA lockout, invoice and PO copies, API rate-limit questions, and seat-license changes. Another 18% are bug reports that need engineering, and 12% are billing disputes requiring a human with account access.
Leadership wants Fin live in six weeks with a published resolution target. The content team is one technical writer at 50% allocation. Marketing owns the help center and refuses a full rewrite before launch. Legal has ruled that Fin must never state contractual refund or cancellation terms. The support lead wants to switch Fin on across all channels at once, arguing it will produce 'real data' faster.
1. Which of the four high-volume topics should the pilot include, and which conversations must be excluded? Explain how your scoping choice affects the number you will report to leadership.
Read the worked answer
Screen each topic against three tests: can Fin answer it from existing content, is it a single-turn information request, and is it free of legal exposure?
- Password/MFA lockout - almost always in scope. Deterministic, article-covered, low risk. The only caveat is verification: if self-serve reset fails, the customer needs a human.
- API rate-limit questions - in scope only if the articles are version-specific and current. Rate limits change by plan; a stale article produces confident wrong answers.
- Invoice/PO copies - this is an action, not an answer. Fin cannot resolve it until a procedure or integration exists. Exclude for now, or include only the 'where do I download my invoice' variant.
- Seat-license changes - exclude. It touches billing state and often contract terms, which Legal has prohibited.
- Bug reports and billing disputes - exclude outright; they need engineering or account access.
Why the scope drives the metric. Resolution rate is a ratio over the conversations Fin was enabled on. If you enable it on everything, the 30% that is structurally unresolvable drags the headline number down and tells you nothing about content quality - you cannot separate 'bad content' from 'out of scope'. Report resolution rate on the in-scope cohort, and separately report handoff latency and CSAT for the excluded cohorts so leadership sees the whole picture rather than one blended figure.
2. Marketing will not rewrite the help center before launch. What is the most defensible content plan for six weeks with one half-time writer?
Read the worked answer
A full rewrite is the wrong objective; coverage of the pilot topics is the objective. Prioritise by conversation volume, not by article count.
- Rank the top 20–30 articles that actually appear in the four pilot topics' transcripts. Those carry most of the resolvable volume; the remaining 870 articles are irrelevant to launch.
- Rewrite those articles from real transcripts. Pull the phrasing customers use and answer in that language, in the first paragraph. Marketing's objection is usually about brand voice, not accuracy - agree a house style sheet once and apply it yourself.
- Resolve contradictions before adding anything. Fin will stitch together multiple sources. If an old pricing page and a new one both exist, archive the old one. Contradiction is a bigger risk than omission.
- Get Legal to approve three or four canonical refusal phrases for refunds and cancellation, and exclude those intents from the audience rather than hoping Fin phrases it well.
- Record known-unanswerable questions and configure the handoff behaviour for them explicitly.
- Test offline by replaying 100 held-back transcripts through the answer preview before touching a live audience. That is the only cheap way to find confident-wrong answers in week four instead of week seven.
3. Critique the plan to launch on all channels simultaneously, and propose a launch sequence with concrete guardrails.
Read the worked answer
The critique. Channels fail differently. Chat conversations are short, synchronous and often ambiguous; email is asynchronous, multi-message, and frequently carries attachments and long threads. Launching both at once doubles the blast radius of a bad content set, and - more importantly - makes attribution impossible: if CSAT drops, you cannot tell whether the cause is chat answer quality, email formatting, or the handoff queue. It also assumes the handoff path is ready on both channels, which it usually is not.
A defensible sequence.
- Week 1–2: chat only, logged-in customers, restricted to the password/MFA and API topics, 10% of eligible traffic.
- Week 3–4: widen to 50%, add the invoice-download variant once the procedure exists. Keep the kill switch and audience rule live.
- Week 5–6: full chat, then email on a small cohort after chat metrics have been flat for two weeks.
Guardrails, agreed in advance with leadership:
- CSAT on Fin conversations drops more than 0.3 below the human baseline.
- Handoff queue time to first human reply exceeds the team's SLA.
- Reopen rate within 7 days on Fin-resolved conversations rises above the pre-launch baseline.
- Any confirmed Legal-sensitive answer.
Any one of these triggers automatic rollback of the audience rule, not a meeting to discuss it. Pre-committing to the rollback removes the argument in the moment.