Claims Triage: Five Agents, Duplicate Calls, and a Router That Jumps the Queue
A regional insurer runs claims triage on a five-agent system in OCI: an intake router, a policy-lookup agent, a coverage-analysis agent, a fraud-signal agent, and a settlement-drafting agent. The router is exposed to 23 tools. Volume is about 40,000 claims a month. Two months after launch, p95 latency is 38 seconds, 34% of tool calls within a session are duplicates, 12% of sessions hit the 8-step orchestration cap, and human escalation is 22% against a 12% target. Traces show the coverage agent calling get_policy_terms three to four times per claim with identical arguments, and the router sometimes dispatching a claim to settlement-drafting before fraud signals return. Each agent keeps its own conversation buffer; there is no shared case state and no per-case cache. The business wants escalation reduced first and latency second. A vendor is proposing a larger model as the primary fix.
1. Diagnose the dominant cause of the escalation and latency problems, and describe the architecture you would move to. Justify each change against the trace evidence.
Read the worked answer
Diagnosis. The traces point to missing shared case state and unconstrained orchestration, not model capability. Identical get_policy_terms calls mean results are not memoized and no agent can see what another already learned. Dispatch to settlement-drafting before fraud returns means ordering is being decided by the router's reasoning rather than enforced by the system. Hitting the step cap in 12% of sessions is the signature of loop-prone control flow with no precondition checks.
Target architecture.
- Introduce a typed case object (claim_id, policy_version, coverage_decision, fraud_score, status, evidence_refs) as the single source of truth. Tools read and write this state instead of appending to per-agent chat buffers.
- Replace LLM-driven routing with an explicit graph/state machine. Where a decision is a lookup (does this claim type need fraud review?), encode it as a rule, not a prompt.
- Add a tool-result cache keyed by (tool name, normalized arguments, case state version). The three duplicate policy lookups collapse to one.
- Enforce the precondition on settlement-drafting in code: fraud_score present and coverage_decision = covered. Prompt instructions are advisory; graph edges are testable.
- Cap each tool at two calls per case, then fall back to the cached result or escalate with a structured summary.
Expectation. Duplicate-call ratio should fall sharply and step counts drop below the cap, which is what actually drives the 22% escalation. Latency improves as a side effect. Measure before adding model capacity.
2. How would you enforce the fraud-before-drafting ordering and bound runaway loops, and why is a prompt instruction insufficient for either?
Read the worked answer
Ordering. Model it as a gate the runtime evaluates: the drafting node is only reachable when fraud_score != null and coverage_decision == covered. If the gate fails, the orchestrator either calls the fraud agent synchronously or returns an awaiting_fraud status. This is deterministic, unit-testable, and observable as a state transition.
A prompt like "always wait for fraud before drafting" fails because it competes with everything else in the context, degrades as the conversation grows, and cannot be verified in a test suite. It also gives no signal when it is ignored.
Loop bounds. Layer three limits:
- Per-tool budget (e.g., 2 calls per case), returning the cached result on the second call.
- Per-case step budget (keep 8, but make hitting it an explicit
escalate_incompleteoutcome that carries the partial state and the reason). - Per-node timeout plus an overall session deadline, so a slow backend cannot consume the budget.
Why this matters for escalation. Today, sessions that exceed the cap likely escalate without a usable summary, which pushes work back to humans. An escalate_incomplete outcome with coverage and fraud fields populated lets a human finish in seconds. Also add idempotency keys on any downstream write so a retry after timeout does not duplicate a settlement record.
3. Define the instrumentation and the experiment that would prove the redesign worked, and assess whether the larger model is a plausible primary fix.
Read the worked answer
Instrumentation. Emit one trace per claim with spans for every tool call carrying case_id, tool name, hashed arguments, cache hit/miss, latency, and outcome. Derived metrics:
- Escalation rate (primary), split into
escalate_incompleteandescalate_low_confidence. - Duplicate-call ratio: calls with identical (tool, args hash) per case.
- Steps per case and share hitting the cap.
- p50/p95 latency per node, to see where the 38 seconds actually goes.
- Containment rate and human edit rate on drafted settlements (quality proxy).
Experiment. Canary 5% of claims for one week, then 25%, comparing against a matched holdout rather than before/after alone, since claim mix shifts. Pre-register the success thresholds: escalation below 15% at 25% rollout, duplicate ratio under 5%, p95 under 25 seconds.
On the larger model. It might improve extraction from messy claim documents and reduce ambiguity-driven extra loops, so it is worth a bounded test. But duplicate calls with identical arguments and premature dispatch are control-flow defects; a better model may route more confidently and thereby dispatch to drafting earlier, not later. Expect a small effect at higher cost and latency. Sequence the architectural fixes first, then evaluate the model swap on the same golden set so the two effects are not confounded.