Agentic Claims Triage: Scaling Tools Without Breaking Latency
Claims operations at a regional insurer runs a 14-tool agentic triage assistant built on a planner-executor pattern. It handles 2,300 conversations per day across web chat and email. The service-level target is p95 latency under 15 seconds, but production traces show 41 seconds at p95 and a 9% loop rate where the agent calls the same lookup tool three or more times. Tool error rate is 4%, mostly from a policy API that times out under load. The human escalation queue has grown 3x, and 22% of escalations are for cases the agent could have resolved. The team wants to add five more tools for fraud checks and payment status, and the contact-center director wants to keep a single monolithic prompt because 'it is easier to maintain.' Monthly inference budget is capped at $18,000, and current spend is $16,400. The agent has no per-conversation step cap and logs only final answers.
1. Given the latency, loop, and tool-error symptoms, what architectural changes would you make before adding the five new tools? Explain the trade-offs.
Read the worked answer
The symptoms point to control-flow and dependency problems, not a tool-count problem. Adding five tools to a monolithic planner increases branching and prompt length, which usually worsens loop and wrong-tool rates. I would first split the planner into a router that classifies intent and a small set of bounded subagents, each with 3-5 tools and an explicit state machine. Give every tool a typed schema, timeout, retry budget, and idempotency key. Cache read-only lookups and run independent calls in parallel; cap the agent at six steps and force escalation on repeat calls. Fix the policy API timeout before scaling: add circuit breaking, a fallback read replica, and a degraded-mode response. Trade-offs: more components add operational overhead and require trace correlation, but they reduce prompt complexity and make failures local. The monolithic prompt is easier to edit but harder to test and observe. Budget: parallelism and caching should reduce token spend; set a per-conversation token cap and monitor cost per resolution. Go/no-go: canary only after loop rate <2% and tool error <1%.
2. Design an evaluation and guardrail plan that would let you canary the revised agent to 10% of traffic.
Read the worked answer
Evaluation must be trace-level, not just final-answer-level. Build a labeled set of 500 historical conversations stratified by intent, including 100 known failure cases. Score task success, tool selection accuracy, argument correctness, steps to resolution, latency, cost, and escalation appropriateness. Add adversarial cases for tool timeouts and ambiguous requests. For guardrails, enforce allow-listed tools per subagent, schema validation, PII redaction before external calls, and a human approval gate for irreversible actions such as payment status changes. Use a policy engine outside the prompt so limits cannot be prompted away. Canary plan: shadow traffic first, then 5% for three days, then 10% for a week, with automatic rollback if p95 latency >15s, loop rate >2%, or task success drops more than 5 points versus control. Compare against the current agent on the same traffic slice, not just offline. Log full traces with tool inputs/outputs, and review 50 canary traces daily. Only expand if the guardrail violation rate is near zero and escalation quality improves.
3. How would you handle irreversible actions such as changing a payment status?
Read the worked answer
For irreversible actions, the agent should not have direct write access. Use a propose-confirm-execute pattern: the agent drafts the action with evidence, a deterministic policy service checks eligibility and limits, and a human or pre-approved workflow confirms. Every action gets an idempotency key and an audit record. If the payment API times out, retry with the same key and check status before resubmitting; never retry a write blindly. Add compensation for partial failures and a dead-letter queue. This adds latency, so keep read-only lookups fast and reserve approval for high-risk actions. Measure escaped defects and approval turnaround. The business trade-off is speed versus control, but irreversible actions warrant the extra step.