AI customer agent pilot for a 2,400-chat e-commerce support desk
Lumine is a direct-to-consumer skincare brand with 11 support agents and about 2,400 chats a month, plus email. Sixty-eight percent of chats are order status, returns, or ingredient questions. Fourteen percent are in French, but the knowledge base is English-only and three return-policy articles were last updated before the January policy change. The current menu chatbot contains 22% of chats and has 61% CSAT; customers complain about loops. The VP of CX wants to launch HubSpot's AI customer agent and reach 70% containment in 60 days, then cut two agent roles. The support lead worries about wrong ingredient advice and refund promises. Order status requires an integration that is not live yet. The team has 140 knowledge base articles, 19 of which are flagged outdated.
1. Assess the 70% containment and 60-day goal. What would you tell the VP, and what would you do first?
Read the worked answer
The 70% containment goal in 60 days is not a readiness plan. Containment measures whether the bot avoided a human, not whether the customer got a correct answer. The current menu bot already contains 22% and still has 61% CSAT, which shows that containment can rise while experience falls. Readiness gaps are concrete: the order-status integration is not live, so the agent cannot answer the highest-volume intent; the French experience has no knowledge parity; the return policy changed and three articles are stale; and ingredient questions can become health questions.
I would tell the VP: we will target containment by intent, with a quality gate. First 30 days: fix the return-policy articles and confirm the order-status integration; baseline chat reasons; pick three low-risk, high-volume intents; and set a stop rule that any intent with accuracy below the agreed threshold is removed. Do not cut two roles before the pilot proves sustained CSAT and repeat-contact results. A realistic first target is containment on the selected intents, not 70% of all chats, and no automation for adverse reactions, refunds, or account changes until there is a reviewed process.
2. Design a pilot that is safe and measurable. Specify intents, guardrails, handoff, and what you would not automate.
Read the worked answer
Pilot scope:
- Order status, only if the integration is live and authenticated data can be returned. If not, do not let the agent guess.
- Returns policy, using the single approved article and a link to the returns flow.
- Basic product usage questions that are answered in the knowledge base.
Guardrails:
- The agent answers only from approved knowledge sources; stale or internal-only articles are excluded.
- Escalate on low confidence, repeated fallback, angry sentiment, adverse reaction, refund or compensation request, account changes, or French until French content is reviewed.
- Handoff passes the transcript, customer identity, intent, and what was already tried. Do not make the customer repeat themselves.
- Log every conversation for weekly transcript review; publish a kill switch and an owner.
What not to automate: medical or adverse-reaction advice, refund promises, price exceptions, account security changes. Measure CSAT by path and repeat contact within seven days; if either degrades, narrow the scope.
3. Which metrics and governance would you use to decide whether to expand?
Read the worked answer
Expansion should be earned on four dimensions:
- Quality: audited answer accuracy by intent, escalation appropriateness, and CSAT for bot-contained versus human-handled chats.
- Efficiency: containment by intent, human handoff rate, handling time saved, and repeat contacts within seven days.
- Risk: complaint rate, adverse-reaction misroutes, and policy exceptions.
- Coverage: languages and intents with current approved content.
Governance: a named content owner for each intent, a weekly review of transcripts and failed searches, a legal/compliance review for claims and returns language, and an incident process for bad answers. I would expand only when the pilot beats the human baseline on CSAT and repeat contact, not merely when containment rises. Agent roles should be redeployed to complex cases and review, not cut on a containment number alone.