Sandbox Star, Production Problem
Lumen Health's patient-support agent passed sandbox testing with 40 scripted utterances and a 92% success rate. In production, over six weeks, containment is 54% against a 70% target, average response latency is eight seconds, and escalations spike between 5 p.m. and 9 p.m. The sandbox used 50 synthetic orders and a web chat widget; production serves 200,000 patients over SMS and mobile web, with Spanish-language users and a long tail of insurance questions. Test cases covered happy-path appointment rescheduling and clinic hours, but not ambiguous requests, after-hours behavior, or handoff failures. Leadership wants to 'turn up autonomy' and remove the human escalation queue to hit the target.
1. Diagnose the gap between the 92% sandbox result and 54% production containment.
Read the worked answer
The sandbox result is not evidence of production readiness because the test distribution is not representative.
- Sample bias: 40 scripted happy-path utterances and 50 synthetic orders cannot represent 200,000 patients and a long-tail insurance domain.
- Channel effects: SMS and mobile web differ from a web widget in message length, formatting, session continuity, and user expectations. SMS users may send fragments; the agent may ask clarifying questions that feel like containment failures.
- Language coverage: Spanish-language users were not tested, so topic classification, retrieval, and response quality may degrade.
- Time-of-day behavior: the 5–9 p.m. escalation spike suggests after-hours clinic or human-queue availability, or a shift in request mix, not necessarily a model failure.
- Missing failure tests: ambiguous requests and handoff failures were untested, so the team did not see how often the agent gets stuck or transfers without context.
- Latency: eight seconds is long enough to cause abandonment and repeated messages, which look like low containment.
The team measured the agent on an easy test set and is now measuring it on the real distribution.
2. Design a testing and monitoring plan before changing autonomy.
Read the worked answer
Build the test set from production reality, then keep measuring online.
Offline
- Stratify sampled production transcripts by intent, channel, language, time of day, and outcome. Include ambiguous, multi-intent, adversarial, and handoff-failure cases.
- Create regression suites for the top intents plus a long-tail sample, and label expected containment, escalation, or refusal.
- Run channel-specific tests: SMS fragmentation, mobile web, and Spanish.
Online
- Canary or A/B rollout by channel or segment; compare containment, escalation reason, repeat-contact rate, CSAT, latency, and transfer-with-context rate.
- Monitor after-hours behavior separately; inspect why escalations spike.
- Add human review of a sample of contained and escalated conversations.
Autonomy
- Change autonomy only after metrics are stable across strata. Define rollback thresholds and an owner. The eight-second latency and the 5–9 p.m. spike should be diagnosed before increasing autonomous action.
3. Evaluate leadership's proposal to remove the human escalation queue to reach 70% containment.
Read the worked answer
Removing the escalation queue would not create containment; it would hide failure and shift risk to patients.
- Containment is not the same as resolution. If the agent cannot resolve a case, removing the queue may cause abandonment or repeated contacts, which could look better or worse depending on the metric but does not help the patient.
- Escalation is a safety valve for ambiguous, urgent, or after-hours requests. The 5–9 p.m. spike suggests a real operational need, not a queue the agent should eliminate.
- Autonomy should be risk-based. Permit autonomous action for low-risk rescheduling and information requests; require human review for insurance eligibility, clinical questions, and exceptions.
- Improve handoffs first. Pass a summary, verified patient context, and reason for escalation so the human can resolve quickly.
A better target is: raise containment where evidence supports it, keep escalation available, and measure resolution and trust, not just the percentage of conversations the agent ends.