Litware: an over-tooled service desk agent

Diagnosis: 22 tool schemas are re-sent on every turn. Beyond cost, the model must discriminate between overlapping tools (three SharePoint search tools, several REST endpoints) on every step, and each misstep compounds across 6.4 turns. This is a selection problem, not a model-quality problem.

Fix in this order

1. Prune. Remove tools that no live task uses; merge the three search tools into one parameterised tool. A smaller, sharper catalogue beats a complete one.
2. Rewrite descriptions. Each description states when to use it, when not to use it, required parameters, an example call, and what an error means. 'Gets user data' invites the wrong call; 'Use only for the calling employee's own HR record; for another employee use tool X' prevents it.
3. Split by domain with connected agents. A triage agent holds 3-4 delegation tools (IT agent, HR agent, facilities agent); each specialist holds only its own tools. This keeps each model context small and is the durable answer as the catalogue grows.
4. Shrink tool outputs. Returning full JSON objects bloats context and degrades every later turn. Return only the fields the agent needs.

The refund incident is separate: a destructive, money-moving action should never be a directly callable tool. Remove it and expose only request_refund_approval, which creates an approval record. Validate amounts server-side - never rely on the model to be careful.

The timeout is a client design fault as much as an agent fault. Holding one HTTP request open for a multi-step run with 6+ model turns cannot work. Change the client to start a run, return a run/thread identifier immediately, and poll or stream status. The thread keeps conversation state, so the work does not have to live in one request.

For genuinely long work, move orchestration to a durable layer (Durable Functions or an equivalent queue-driven workflow) that survives client disconnects and reports progress; the agent's job becomes deciding what to do, not staying alive while it happens.

Approvals - do not block on a human

• The agent calls a request_*_approval tool that writes an approval record and returns {status: pending, approvalId}.
• The run ends deliberately; the user is told what is pending.
• An approver acts in the approval UI or by email.
• A new run resumes on the thread with the decision in context and completes or abandons the action.

This keeps the approval auditable (who approved, when, against which request) and keeps latency bounded. Do not implement approval as a tool that polls until a human clicks.

Interim feedback also matters operationally: a user who sees 'checking your ticket and VPN state' does not retry and create duplicate tickets, which is a hidden driver of the 3.5× cost.

Where the cost actually comes from: 6.4 turns × a growing transcript including full tool payloads. The fix is turn count and context size, not a cheaper model - a cheaper model with the same confusion may increase turns and cost more.

Levers

• Summarise or drop older turns; keep the last few verbatim plus a running summary.
• Trim tool responses to the fields needed.
• Cap the tool-call loop in your orchestration and return a partial result plus escalation instead of looping.
• Cache tool results within a thread (ticket state fetched twice is paid twice).
• Route simple intents through a single-call, no-tool path: a classifier sends password-reset and status questions to a deterministic handler and reserves the agent for multi-step work.

Making changes safe

• Build a regression set of 60-100 realistic multi-step tasks with expected tool sequences and outcomes.
• Run it on every tool schema, instruction or model change; track task success, wrong-tool rate, turns and cost per run.
• Log every run and tool call with IDs so a wrong-action incident can be reconstructed - the refund incidents were only detectable because the calls were logged.

Governance: require a description, owner, example call and a regression task with every new tool. Otherwise the catalogue regrows to 22 and the problem returns.