Claude Certified Architect - Foundations common mistakes
Claude Certified Architect - Foundations · Understand the traps and how to avoid them.
Treating the model's context window as a fixed hard limit and assuming any conversation that fits is equally reliable, rather than recognizing degradation and 'lost in the middle' effects on long inputs.
Candidates memorize a single context-window number per model and assume recall is uniform across the window, so they miss questions about placing critical instructions or retrieved evidence near the start or end of long contexts.
What to do instead
Study how position affects recall in long contexts and the principle of putting the most important instructions and evidence at the beginning or end. Practice reasoning about context budget trade-offs between history, retrieved documents, and instructions.
Designing a tool-use loop that calls tools but never validates arguments, never handles tool errors, and never sets a termination condition, producing infinite or runaway agent loops.
Candidates focus on the happy path of 'model requests tool, tool returns result' and forget production concerns like schema validation, error feedback to the model, iteration caps, and stopping criteria.
What to do instead
For every agentic design question, explicitly include: strict input schema validation, structured error messages returned to the model, a maximum iteration/turn limit, and a defined success/termination condition.
Writing system prompts that pile on vague adjectives ('be helpful, accurate, and safe') instead of concrete behavioral rules, output format specs, and explicit refusal/edge-case instructions.
Candidates equate longer or more emphatic prompts with better prompts and underestimate how ambiguity in instructions produces inconsistent outputs that are hard to evaluate.
What to do instead
Practice rewriting prompts into explicit, testable requirements: role, task, constraints, output schema, examples, and handling for ambiguous or out-of-scope inputs. Evaluate whether each sentence is verifiable.
Confusing prompt engineering with fine-tuning or RAG, and recommending the wrong intervention (e.g., proposing fine-tuning to inject frequently changing factual knowledge).
Candidates learn the techniques in isolation and do not build a decision framework mapping problem types (knowledge freshness, format compliance, tone, latency, cost) to the right solution.
What to do instead
Build a comparison table: prompting for behavior/format, RAG for dynamic or proprietary knowledge with citations, fine-tuning for stable style/format at scale. Match each exam scenario to the cheapest adequate intervention.
Assuming adding a retrieval step automatically improves answers, without addressing chunking strategy, embedding quality, reranking, or grounding/citation of retrieved content.
Candidates treat RAG as a single checkbox and skip the pipeline details that determine whether retrieved context is relevant and whether the model actually uses it faithfully.
What to do instead
Study each RAG stage independently: chunk size/overlap trade-offs, embedding model choice, hybrid/keyword plus vector retrieval, reranking, and prompting the model to cite or abstain when evidence is insufficient.
Designing safety guardrails as a single input filter, ignoring output-side checks, tool-permission scoping, and defense against prompt injection from retrieved or tool-returned content.
Candidates picture safety as blocking bad user prompts and forget that untrusted content entering via RAG or tools can hijack the model, and that outputs can leak data or take unsafe actions.
What to do instead
Layer defenses: input validation, treating retrieved/tool content as untrusted data (not instructions), least-privilege tool scopes, human approval for high-impact actions, and output filtering. Articulate defense-in-depth in answers.
Proposing an evaluation plan based only on 'eyeballing outputs' or a single accuracy metric, with no golden dataset, no rubric, and no LLM-as-judge calibration.
Candidates underinvest in evaluation methodology and do not know how to make automated grading trustworthy, so they miss questions about judge bias, rubric design, and regression testing.
What to do instead
Learn to specify: a curated eval set with representative and adversarial cases, explicit rubrics, LLM-as-judge with human spot-checks and position-bias mitigation, and running evals as regression tests on every prompt/model change.
Ignoring operational realities in deployment answers: no handling of rate limits, retries with backoff, timeouts, streaming, token/cost budgeting, or latency trade-offs.
Candidates answer architecture questions at the whiteboard level and omit the failure handling and cost controls that distinguish a production system from a prototype.
What to do instead
For any deployment scenario, enumerate: idempotent retries with exponential backoff and jitter, timeout and fallback behavior, streaming for perceived latency, token accounting and caching, and monitoring/alerting on error rates and cost.
Choosing the largest or most capable model by default instead of matching model tier to task complexity, latency, and cost requirements.
Candidates equate 'best' with 'biggest' and do not consider that many tasks (classification, extraction, routing) are well served by smaller, faster, cheaper models with proper prompting.
What to do instead
Practice a model-selection framework: define quality bar, latency budget, and cost ceiling per task, then start with the smallest model that passes evals and escalate only when evaluations show it is needed.
Treating observability as basic logging of requests, without capturing prompts, model versions, tool calls, token usage, and trace context needed to debug and reproduce failures.
Candidates think of observability as generic application logging and miss the LLM-specific signals (prompt/response pairs, sampling, traces across multi-step agent runs) required for diagnosis.
What to do instead
Design traces that capture full prompt/response, model and prompt version, tool invocations and results, token counts, latency, and a correlation ID across agent steps, with PII redaction and sampling for cost control.