AI-500 cheat sheet

Orchestration patterns: sequential pipeline (fixed order), concurrent/fan-out-fan-in, hierarchical supervisor-over-workers, and decentralized handoff (peer-to-peer routing).

Supervisor pattern: a coordinator agent decomposes the task, assigns subtasks to specialist agents, and aggregates results; specialists never call each other directly.

Handoff pattern: control transfers to the peer agent best suited for the next step; the active agent changes, unlike supervisor where control returns to the coordinator.

Single-responsibility rule: one agent = one role, one tool set, one instruction set; broad agents reduce routing accuracy and increase prompt conflicts.

Stateless agents plus an external conversation/state store scale better than in-process memory; state must be serializable to survive restarts.

Cap every loop: max handoffs, max turns, and max tool calls per agent are required to prevent infinite agent-to-agent ping-pong.

Use a deterministic workflow (code-defined graph) when steps are known; use LLM-driven routing only when the path is genuinely dynamic.

Agent-to-agent contracts should be typed (JSON schema for request and response) so a failing agent surfaces as a validation error, not silent garbage.

Route by capability registry, not by model guesswork: each agent publishes name, description, and tools; the router selects on that metadata.

Azure AI Foundry Agent Service provides managed agents with threads, runs, and tool calling; connected agents implement supervisor/handoff topologies without custom orchestration code.

Tool types include function calling (client-executed), Azure AI Search, Bing grounding, Code Interpreter, File Search, and OpenAPI-spec actions.

Function-calling flow: model returns tool_calls with name and JSON arguments, the client executes, then submits tool outputs keyed by tool_call_id in the next turn.

Threads persist message history server-side; a run is the execution of an agent against a thread and exposes statuses such as queued, in_progress, requires_action, completed, failed.

requires_action status means the run is blocked awaiting tool output submission; it is not an error and must be resumed, not retried from scratch.

Connected agents are declared per-agent with a name, description, and type; the description is the routing signal the parent uses to delegate.

Use the SDK/portal for prototyping but pin agent versions and instructions in source control (IaC/Bicep or SDK scripts) for reproducible deployments.

Streaming (event-based run steps) is required for low-latency UX; non-streaming polling adds latency equal to the full run duration.

Evaluate at two levels: end-to-end task success and per-agent step quality; a passing end-to-end run can hide a failing intermediate agent.

Core metrics: groundedness, relevance, coherence, fluency, task completion, and intent resolution; retrieval adds recall and precision at k.

Golden datasets should include adversarial, ambiguous, and out-of-scope inputs; a set of only happy-path cases inflates scores.

LLM-as-judge requires a rubric, a fixed judge model version, and temperature 0; changing the judge invalidates comparisons across runs.

Token cost = prompt tokens + completion tokens across all agents and turns; multi-agent fan-out multiplies cost by the number of delegated calls.

Latency budget is cumulative: orchestrator planning + N sequential agent calls + tool round-trips; parallelize independent branches to cut wall-clock time.

Trace with OpenTelemetry-style spans per agent and tool call; correlate by conversation/thread ID to reconstruct a multi-agent run.

Monitor production with metrics for tool error rate, handoff count per request, token spend per conversation, and p95 latency; alert on drift from baseline.

Optimization levers: trim instructions, cache static context, reduce handoffs, cap max tokens, and route simple intents to a cheaper model.

Use managed identity for agent-to-service auth; avoid embedding keys in agent instructions or tool definitions, which are readable by the model.

Least privilege per agent: scope each agent's tool permissions to only the resources its role needs; a shared super-credential defeats agent isolation.

Prompt injection defense: treat all retrieved and tool-returned content as untrusted data, never as instructions; enforce allow-lists on tool arguments.

Excessive agency risk: constrain tool parameters server-side (e.g., query filters, write allow-lists) rather than relying on the prompt to behave.

Content safety and prompt shields run as pre/post filters; log blocked events for audit and tune thresholds against false positives.

Governance: version agents and instructions, keep an approval gate before production promotion, and record evaluation scores with each release.

Data boundary: keep secrets in Key Vault, use private networking/private endpoints for Foundry and Search, and disable public egress where required.

Deployment: blue-green or canary with traffic split, rollback on regression in task success or error rate; agents are versioned artifacts, not mutable config.

Audit every tool invocation with caller identity, arguments hash, and outcome; RBAC on the control plane is separate from agent runtime permissions.