Analytics, Evaluation and Continuous Improvement
What this domain tests
This domain tests whether you can measure assistant quality and improve it without guessing. It covers operational analytics, conversation review, offline test sets and controlled iteration.
Core rules
- Define success before launch. Useful measures include containment, resolution, escalation, fallback rate, abandonment, action completion, knowledge answer rate and customer satisfaction.
- Containment is not resolution. A bot can contain a user by trapping them; pair containment with resolution and repeat-contact measures.
- Segment metrics by channel, action, knowledge source, language and release. A flat average can hide a failing segment.
- Review transcripts and label failure modes: no match, wrong action, bad retrieval, wrong tool result, poor wording, handoff failure.
- Build a regression set of representative utterances and expected outcomes. Re-run it after prompt, model, content or flow changes.
- Compare versions with a baseline and a clear hypothesis. Change one major variable at a time where possible.
Worked example
After a publish, fallback rate rises from 8% to 15% overall. Segmentation shows the increase is concentrated in the “change address” action on the voice channel. Transcripts show users say “flat number” but the slot expects “apartment.” The team adds synonyms and a re-ask example, then re-runs the regression set. Reasoning: the aggregate metric hid a channel-specific vocabulary gap; the fix targets the failure mode rather than adding generic fallback wording.
Traps
Optimizing containment alone; using an unlabeled “looks good” review; changing prompts and retrieval at the same time; ignoring low-volume but high-risk actions; and failing to keep a baseline after each release.