Parallel Agents Blowing the Token Budget
A financial-analysis multi-agent system runs on Azure AI Foundry. A supervisor agent delegates to four research agents (filings, news, analyst notes, market data), each of which can call tools and delegate further. Median task latency is 41 seconds and median cost is $0.94 per request; the product target is under 15 seconds and $0.30. Telemetry shows the supervisor re-sends the full conversation history to each of the four agents, each agent then re-sends that history plus its own tool results to the synthesis step, and the filings agent delegates to a sub-agent that re-reads the same history a fourth time. Average input tokens per request are 190k against 6k output tokens. Two of the four agents return 'no relevant findings' on 70% of requests.
1. Diagnose where the cost and latency are going and propose the highest-leverage changes, ordered by expected impact.
Read the worked answer
The token profile (190k in / 6k out) says this is an input-amplification problem, not a reasoning problem. History is being duplicated at every hop: supervisor→4 agents (4 copies), each agent→synthesis (4 more copies plus tool output), and one sub-delegation adds a fifth. That is roughly 9–10 traversals of a growing context.
Highest-leverage changes, in order:
- Pass scoped context, not full history. Each specialist should receive only the sub-question and the minimum facts it needs, not the entire conversation. This alone typically removes the majority of the 190k.
- Synthesize once, centrally. Agents should return compact structured findings (a few hundred tokens), and the supervisor should synthesize from those summaries. Do not let agents re-send history to a synthesis step.
- Cap delegation depth. A sub-agent re-reading the same history is pure waste; enforce a max depth of 1–2 and forbid re-reading parent context.
- Skip the agents that usually return nothing. With 70% empty results on two agents, gate them behind a cheap relevance check or run them only when the question's intent matches. An agent that returns nothing 70% of the time is a latency and cost tax on every request.
Expect the biggest wins from 1 and 2 (context volume), then 4 (fewer parallel branches = lower tail latency). Latency at 41s is dominated by parallel fan-out plus serial synthesis; trimming branches and shrinking prompts attacks both.
2. A teammate proposes switching all four agents to a smaller, cheaper model to hit the $0.30 target. Assess this against the diagnosis.
Read the worked answer
It is treating the symptom. If input volume is the problem, a cheaper per-token model reduces cost but does nothing for the 41-second latency, and it risks degrading the quality of the filings and analyst-notes agents, which do the most demanding extraction. You would hit the cost number while missing the latency target and possibly worsening accuracy.
Better sequencing:
- Fix the context duplication first - that reduces both cost and latency and is model-agnostic.
- Then right-size per agent: a small model is appropriate for the intent-classification/relevance gate and for the two agents that usually return nothing; keep stronger models for filings extraction and final synthesis.
- Measure after each change; the $0.30 target may be met by context reduction alone, making the model downgrade unnecessary.
The general principle: match model capability to task difficulty per role, and fix structural waste before paying for a downgrade in quality. Downgrading a model is a one-way quality trade; removing duplicated tokens is free.