Misleading containment and a missing start-service capability
Harbor Utilities' assistant dashboard reports 78% containment. Customer satisfaction is 2.9/5, and 41% of contained conversations end with the assistant saying 'Is there anything else I can help with?' with no confirmed resolution. The top unanswered phrase is 'start service' with 1,900 hits; the team has no action for it. Average handle time for escalated calls is 11 minutes, and repeat contacts within seven days are 29%. The product owner wants to add 50 new actions, raise the LLM temperature to make answers friendlier, and declare success at 85% containment. There are 120 logged conversations per day, and only two reviewers. The compliance team is concerned that the friendly prompt may produce promises the utility cannot keep.
1. Critique the success metric and interpret the dashboard. What does containment hide?
Read the worked answer
Containment here means did not escalate, not resolved. The 41% ending with 'Is there anything else I can help with?' and no confirmed resolution is the key: the assistant closed politely without completing a task. A 29% repeat-contact rate within seven days confirms unresolved issues returning. CSAT 2.9/5 is consistent with deflection, not success. The top unanswered phrase 'start service' with 1,900 hits is a missing capability, not a phrasing problem. A high containment number can be manufactured by refusing to escalate or by giving generic answers. Replace the headline metric with effective containment: contained and resolved, plus repeat-contact rate, task completion, CSAT, and escalation appropriateness. Segment by intent: high containment on rate FAQs may be good; high containment on start service is bad if the customer still has no service. Also check whether the assistant is ending conversations prematurely to keep containment high.
2. Design a prioritization and evaluation approach with only two reviewers.
Read the worked answer
With two reviewers, do not try to read all 120 daily conversations. Use stratified sampling: 20 random conversations per week, all low-CSAT conversations, all start-service conversations, and all escalations. Label resolution (confirmed task completion), failure reason, and whether a human was needed. Build a 200-conversation golden set, with 50 double-labeled to calibrate. Prioritize by volume times failure rate times business risk: start service first because of 1,900 hits and likely regulatory or revenue impact; then billing disputes and outage reporting. Add the action with a backend extension, not just an FAQ. Evaluate pre and post with a holdout and review 30 transcripts per change. Use an LLM judge only after it matches human labels on the calibration set. Two reviewers can sustain this if the sample is targeted rather than exhaustive. Track the backlog by evidence, not by the product owner's list of 50 actions.
3. Evaluate the proposed changes (50 actions, higher temperature, 85% target) and recommend guardrails.
Read the worked answer
Reject the package as stated. Fifty actions without a failure map will spread effort; sequence by the top intents and resolution gaps. Raising temperature to sound friendlier increases variability and the chance of unauthorized promises; keep temperature low (0-0.2 for policy, up to 0.4 for style) and achieve friendliness through prompt style and examples, not randomness. An 85% containment target rewards deflection; set a resolution target and guardrails: CSAT at least 4.0, repeat contact below 15%, hallucination or unauthorized-promise rate below 2%, and escalation appropriateness at least 90%. Add a no-commitment-without-backend-check rule for service dates, rates, and eligibility. Run a canary and stop if complaints rise or compliance flags a promise. The product owner should report effective containment, not raw containment. The compliance concern is valid: a friendlier tone can make an unsupported promise sound more credible, so grounding and approval matter more than style.