Evaluate, optimize, and monitor multi-agent solutions
This domain rewards candidates who can separate evaluation (offline, pre-release quality measurement) from monitoring (online production health). For evaluation, know the metric families: groundedness, relevance, coherence, fluency, retrieval quality, task success, and tool-call accuracy. Understand that you need curated test datasets including golden answers, edge cases, and adversarial inputs, and that you evaluate the system end-to-end as well as per-agent - a strong orchestrator cannot compensate for a weak specialist. Know the role of AI-assisted evaluators (model-graded scoring) and their limits: calibrate them against human labels, and watch for evaluator bias such as preferring longer answers.
For optimization, the levers are prompt/instruction refinement, model selection and routing to cheaper models for simple steps, retrieval tuning (chunking, hybrid search, reranking), caching, and reducing agent hops. For monitoring, track latency per agent, token consumption and cost, tool error rates, escalation and fallback rates, and safety/blocked-response counts. Traces spanning agents are essential - you need correlation ids to reconstruct a multi-agent run.
Traps: assuming a single aggregate score is sufficient (it hides per-agent regressions); treating online monitoring as a replacement for offline evaluation; and ignoring cost/latency regressions when accuracy improves. Another trap is evaluating with the same data used for prompt tuning, producing inflated scores.
Rule of thumb: hold out an untouched evaluation set, gate releases on per-agent and end-to-end thresholds, and alert in production on tool error rate and latency percentiles rather than average latency.
Worked example: a support system scores 0.91 end-to-end but the refund agent's tool-call accuracy is 0.62. Per-agent evaluation catches it; fix tool schema and add few-shot examples, then re-run the held-out set before promoting.