AI-500 common mistakes
AI-500 · Understand the traps and how to avoid them.
Designing every multi-agent solution with a single orchestrator that calls all specialist agents synchronously in one long request chain, then defending it as 'centralized control'.
Candidates equate 'orchestration' with 'one controller' and overlook patterns like hierarchical, peer-to-peer, blackboard, or event-driven handoff that better match latency, fault isolation, and independent scaling requirements.
What to do instead
Before choosing a topology, map the task decomposition, who owns shared state, and which steps must be sequential versus parallel; then justify the chosen pattern (supervisor, hierarchical, decentralized handoff) against those constraints rather than defaulting to a single synchronous chain.
Giving every agent in the system the union of all tools and permissions 'so any agent can handle anything', rather than scoping capabilities per agent role.
It feels simpler and avoids runtime failures when an agent is asked to do something outside its lane, but it violates least privilege and makes blast radius and auditing impossible to reason about.
What to do instead
Define an explicit capability contract per agent (tools, data scopes, allowed downstream agents) and design the orchestrator to route to the agent that owns the capability; treat missing capability as a routing decision, not a reason to broaden permissions.
Treating the agent handoff contract as free-form natural language between agents instead of a typed, versioned message schema.
Natural-language handoffs look flexible in prototypes and demo well, but they cause silent misinterpretation, break when one agent's prompt changes, and make failures untraceable.
What to do instead
Specify structured request/response schemas (fields, required context, error codes, correlation IDs) for each edge in the agent graph, version them, and validate messages at the boundary before the receiving agent reasons over them.
Implementing agents in Azure as tightly coupled in-process function calls in a single app, then claiming the design is 'multi-agent' and scalable.
Candidates conflate logical agent roles with runtime isolation and skip decisions about hosting (container apps, functions, agent services), state stores, and independent deployment.
What to do instead
Decide per agent whether it needs independent scaling, independent deployment, or isolation, and map that to concrete Azure hosting and messaging choices (queues, service bus, durable orchestration) with explicit state persistence outside the agent process.
Relying on conversational memory held only in the model context window for shared state across agent turns, with no durable store.
It works in short demos, so candidates never design for restarts, retries, or long-running workflows, and the solution loses state or duplicates work under real failure conditions.
What to do instead
Externalize shared state to a durable store with an explicit schema, make agent steps idempotent using correlation and idempotency keys, and design checkpoint/resume behavior for long-running orchestration.
Wiring tool and model calls with default retry/timeout behavior and no handling for partial failure in a multi-step agent chain.
Candidates focus on the happy-path reasoning loop and treat infrastructure faults as out of scope, so a single transient failure cascades into duplicated side effects or a stuck workflow.
What to do instead
Define per-call timeouts, bounded retries with backoff, circuit breaking, and compensating actions for steps with side effects; test the chain against injected failures (tool timeout, throttling, partial completion) and confirm the workflow converges to a known state.
Evaluating multi-agent output only by eyeballing a few end-to-end transcripts instead of measuring per-agent accuracy and end-to-end task success against a defined dataset.
End-to-end transcripts hide which agent caused a failure, and small samples give false confidence; candidates skip building an evaluation set because it feels like extra work.
What to do instead
Build a labeled evaluation dataset covering task success, per-agent routing correctness, tool-call correctness, and handoff fidelity; run it on every change and report metrics per agent and end to end so regressions are localized.
Optimizing latency or cost by shortening prompts and dropping context without measuring the effect on task success and handoff quality.
Candidates treat token reduction as a pure win and tune against cost dashboards rather than quality metrics, so accuracy silently degrades in edge cases.
What to do instead
Change one variable at a time and evaluate against the same dataset, tracking quality, latency, and cost together; only accept an optimization when quality metrics stay within an agreed threshold.
Monitoring only infrastructure metrics (requests, errors, CPU) and not agent-level behavior such as loop detection, tool misuse, or degraded routing.
Standard observability tooling reports service health, not reasoning health, so candidates assume green dashboards mean a healthy multi-agent system.
What to do instead
Instrument traces with correlation IDs across agent hops, log routing decisions and tool calls, and alert on agent-specific signals (repeated tool loops, fallback rate, handoff rejection rate, token spikes) alongside infrastructure metrics.
Securing the multi-agent system by putting credentials in agent configuration or prompts and treating the orchestrator as implicitly trusted by downstream agents.
Candidates secure the outer API boundary and assume internal calls are safe, missing that agents can be induced to misuse tools or escalate via another agent.
What to do instead
Use managed identities and a secret store for credentials, enforce authorization at each agent and tool boundary rather than only at the edge, scope tokens per agent, and log and review privileged tool invocations with human approval for high-impact actions.