AB-620 common mistakes
AB-620 · Understand the traps and how to avoid them.
Designing an agent solution without first writing down the business outcome, success criteria, and constraints, then jumping straight to choosing a model or agent topology.
Candidates treat design questions as product-selection questions, so they pick a fashionable pattern (multi-agent orchestration, autonomous tool use) before establishing what the workflow must achieve and what limits apply.
What to do instead
For every design scenario, first state the goal, measurable success criteria, data sensitivity, latency/cost budget, and human-oversight needs; only then select a pattern (single agent, deterministic workflow, RAG, or classic automation) and justify it against those constraints.
Choosing a multi-agent or fully autonomous design for a task that a single agent or a fixed deterministic workflow would satisfy more reliably.
Multi-agent designs sound more capable, and candidates conflate complexity with quality, ignoring added failure surface, coordination cost, and debugging difficulty.
What to do instead
Apply the simplest design that meets the requirements; reserve multi-agent decomposition for genuinely separable subtasks with clear handoffs, and prefer deterministic orchestration when the steps are known in advance.
Defining agent instructions and tool schemas vaguely (e.g., 'search the knowledge base' or untyped parameters), then expecting the agent to behave consistently.
Candidates focus on the happy-path narrative and underestimate how much model behaviour depends on precise instructions, explicit tool descriptions, parameter schemas, and stated boundaries.
What to do instead
Write instructions that specify role, scope, step order, refusal conditions, and output format; give each tool a descriptive name, typed parameters with enums where possible, and clear guidance on when and when not to call it.
Building an agent loop with no termination condition, iteration cap, or fallback path when a tool call fails or returns nothing useful.
Candidates validate only successful runs and do not reason about repeated tool calls, empty results, or ambiguous model output, which produce loops or silent failures.
What to do instead
Design explicit exit criteria, a maximum step or retry count, error handling per tool, and a defined escalation or graceful-degradation path; describe these in any workflow design answer.
Treating retrieval as a plug-in step without designing chunking, metadata, refresh, and relevance thresholds, then blaming the model when answers are wrong.
Candidates assume 'add a vector store' equals grounded answers and skip decisions about document granularity, filtering, and how stale or conflicting content is handled.
What to do instead
Design retrieval deliberately: choose chunk size and overlap for the content type, attach metadata for filtering and citation, define indexing/refresh cadence, set similarity or reranking thresholds, and specify behaviour when no relevant content is found.
Connecting an agent to backend systems with broad credentials and no scoping, validation, or confirmation step for write operations.
Candidates concentrate on making the integration work and treat authorization, input validation, and irreversible actions as afterthoughts.
What to do instead
Give each integration least-privilege access, validate and sanitize parameters before execution, require human confirmation for destructive or high-impact actions, and log every call with the identity and inputs used.
Evaluating an AI solution only with a handful of manual happy-path prompts and declaring it ready.
Candidates rely on informal spot checks because building a representative evaluation set takes effort, so edge cases, regressions, and quality drift go undetected.
What to do instead
Define an evaluation dataset covering typical, edge, adversarial, and refusal cases with expected outcomes; measure groundedness, task success, and safety metrics; rerun the suite on every prompt, model, or data change.
Monitoring only availability and latency while ignoring output quality, token cost, tool-failure rates, and user feedback signals.
Candidates map monitoring to traditional uptime metrics and overlook that an AI system can be 'up' while producing degraded or unsafe answers.
What to do instead
Instrument the full pipeline: log prompts, retrieved context, tool calls, token usage, latency, error and refusal rates, and user feedback; define quality and cost thresholds with alerts and a review loop for sampled outputs.
Handling security only at the network or identity layer while leaving prompt injection, data leakage, and unsafe tool use unaddressed.
Candidates apply conventional app-security thinking and do not treat model input, retrieved content, and tool outputs as untrusted data that can influence agent behaviour.
What to do instead
Treat all external content as untrusted, separate instructions from data, constrain tool permissions, filter and validate model outputs, apply content-safety controls, and test with injection and exfiltration scenarios.
Describing deployment as 'publish the agent' without specifying environment promotion, versioning, rollback, and governance controls such as logging, access review, and data-handling policy.
Candidates conflate a working prototype with a production release and omit the operational and compliance scaffolding the exam expects.
What to do instead
Define separate dev/test/prod environments with promotion gates, version prompts, models, and configurations, plan rollback, and specify audit logging, access control, retention, and responsible-AI review as part of the deployment design.