Invoice Exceptions Agent: 41 Auto-Generated Tools and a Duplicate Posting
A finance-operations agent handles vendor invoice exceptions. Its tool catalog was auto-generated from 41 OpenAPI operations, including near-duplicates (get_invoice, get_invoice_by_id, search_invoices) and one write tool, post_journal_entry. Across a three-week pilot of 6,200 tasks: wrong-tool selection occurred in 29%, argument errors in 11% (mostly date format and a missing cost center), and on two occasions a client timeout caused the agent to retry post_journal_entry, creating duplicate journal entries that were reversed manually. The agent calls get_vendor before every line item, adding four to six calls per invoice. Auditors require human approval for any posting above $10,000. The engineering team's proposal is to add more tools so the agent can look up cost centers itself.
1. Redesign the tool surface. Explain how catalog size, naming, and schema design each contribute to the 29% selection error and the 11% argument error rate.
Read the worked answer
Mechanism. Selection accuracy degrades as the catalog grows and as tools become semantically overlapping. Three functions that all return an invoice are indistinguishable to a model choosing from descriptions alone; it will pick one at random-ish and sometimes get a different shape back. That is the 29%. Argument errors are a schema problem: free-text date parameters and optional cost centers invite the model to guess a format. Enums and examples fix more of this than instructions do.
Redesign.
- Collapse to about eight task-shaped tools:
find_invoices(criteria),get_invoice_detail(id, include_lines),get_vendor(id),get_cost_center(code),check_budget(...),propose_journal_entry(...), plus read-only lookup helpers. One way to retrieve an invoice, not three. - Make
get_invoice_detailreturn header plus lines plus vendor in one call. That removes the four-to-six repeatedget_vendorcalls and a whole class of ordering bugs. - Schemas: ISO-8601 dates with a pattern and an example, cost center as a validated string with a lookup fallback, explicit required fields, documented error semantics, and a
when not to use this toolline in each description.
On adding cost-center tools: adding lookups reduces argument errors only if the agent reliably calls them before the write. Better to make the write tool validate and return a structured missing_cost_center error naming valid options, so the correction loop is deterministic.
2. Design the write path so a timeout cannot create a duplicate entry, and specify how the $10,000 human approval requirement is enforced.
Read the worked answer
Idempotency. The client (the agent runtime, not the model) generates an idempotency key per logical posting - a hash of invoice_id, line set, amount, and period - and sends it with every attempt. The posting service stores the key with the resulting journal id and returns the same result for a repeat. On timeout, the agent's first action is get_posting_status(idempotency_key), not a retry. The model never constructs the key; if the model could, a rephrased retry would produce a new key and defeat the mechanism.
Do not blind-retry non-idempotent writes. Retries belong to the runtime with backoff, capped at two, and only for transport errors - never for 4xx responses.
Approval gate. For amounts above $10,000 the tool returns approval_required with a pending transaction id and does not post. Approval happens in a separate interface authenticated as a human approver; the agent cannot call an approve endpoint. The runtime resumes the case when approval arrives, reusing the same idempotency key. Also enforce a per-session cumulative posting cap so a loop of small postings cannot bypass the threshold.
Audit. Every posting record carries the requester identity, the approval id, and the model/tool version, so a reversal can be traced to a specific decision path.
3. How would you evaluate tool selection offline before re-releasing, and which metrics would you gate on?
Read the worked answer
Build a task-level eval set. 400–600 realistic exception scenarios, each labeled with the expected tool sequence and expected arguments. Run them against mocked APIs in a sandbox so no real postings occur, and score:
- Tool-selection accuracy (exact expected tool at each step).
- Argument correctness: exact match for ids and enums, normalized match for dates and amounts.
- Unnecessary-call rate, which is where the
get_vendorpattern shows up. - Task success end-to-end, judged by final state rather than transcript.
- Duplicate-write count, gated at zero - any duplicate is a release blocker regardless of other scores.
Diagnostics. Keep a per-tool confusion matrix. If get_invoice and search_invoices still confuse each other after consolidation, the descriptions are not yet distinct - fix the tool, not the prompt.
Release process. Run the suite in CI on every change to tool schemas or descriptions; schemas are code and should be reviewed like code. Canary 5% of traffic with a shadow comparison of proposed vs. taken tool calls. Track production metrics that mirror the offline ones: wrong-tool corrections, argument-error rate, approval-gate hits, and postings reversed within 24 hours.
Prompt changes alone are not a fix. A prompt that says "use the right tool" cannot compensate for three functions with the same contract.