Claude Certified Architect - Professional cheat sheet

Decompose a solution into five layers: client interface, orchestration, model invocation, retrieval/tools, and persistence; each layer has its own failure and scaling profile.

Stateless orchestration with externalized conversation state (session store) is required for horizontal scaling; in-process memory pins requests to a single instance.

Choose synchronous request/response when latency budget is under ~10s and work is single-turn; choose asynchronous queue plus polling or webhook when work exceeds the client timeout or is long-running.

Idempotency keys on every mutating tool call prevent duplicate side effects during retries; derive the key from the logical request, not the attempt.

Separate the control plane (config, prompts, model routing, policy) from the data plane (inference, retrieval) so prompt and model changes ship without redeploying serving infrastructure.

A context window is a hard architectural constraint: reserve budget for system prompt, retrieved chunks, tool schemas, conversation history, and output tokens before sizing retrieval depth.

Multi-tenant designs must isolate data at the retrieval layer (per-tenant indexes or metadata filters enforced server-side), not only at the prompt layer.

Design for graceful degradation: define fallback behavior for model timeout, rate limit, and retrieval miss (cached answer, smaller model, or explicit uncertainty response).

Match model tier to task difficulty: small/fast models for classification, extraction, and routing; frontier models for multi-step reasoning, ambiguous synthesis, and code generation.

Cost per request = (input_tokens × input_price) + (output_tokens × output_price); output tokens typically cost 3-5x input tokens, so verbosity control is a primary cost lever.

Prompt caching reduces cost and latency for stable prefixes; place invariant content (system instructions, tool definitions, long documents) first and variable content last.

Few-shot examples improve format adherence and edge-case handling but consume context; 3-5 diverse examples usually outperform many near-duplicate ones.

Chain-of-thought or explicit reasoning scaffolds raise accuracy on multi-step tasks at the cost of output tokens; disable for simple lookups.

Structured output (JSON schema or strict tool schema) is more reliable than prose instructions for machine-consumed responses; validate and repair on parse failure.

Context management strategies: summarization of old turns, sliding window with pinned system prompt, and retrieval of relevant history instead of full replay.

Position matters: models attend more reliably to the beginning and end of long contexts, so place the critical instruction or question after the bulk of retrieved content.

Tool definitions need a precise name, description, and typed JSON schema; ambiguous descriptions are the leading cause of wrong-tool selection.

The agent loop is: model emits tool call → runtime validates and executes → result returned as a tool message → model continues; cap iterations to prevent runaway loops.

Prefer parallel tool calls when sub-tasks are independent; serialize when later calls depend on earlier outputs.

Pattern selection: single prompt for one-step tasks; routing for distinct input classes; orchestrator-worker for decomposable subtasks; evaluator-optimizer for quality-critical drafts.

Human-in-the-loop approval gates belong on irreversible or high-blast-radius actions (payments, deletions, external sends), not on read-only lookups.

Tool errors must be returned to the model as structured error messages, not raised as exceptions that terminate the loop; the model can often recover.

Idempotent, narrowly scoped tools with least privilege beat a single general-purpose tool with broad permissions.

Agent state should be explicit and inspectable (plan, completed steps, observations) so failures can be resumed rather than restarted.

Chunking trade-off: small chunks (~200-400 tokens) improve precision, large chunks (~800-1200) preserve context; overlap of 10-20% prevents boundary loss.

Hybrid retrieval (dense vector + lexical BM25) outperforms either alone, especially for exact identifiers, product codes, and rare terms.

Reranking a top-k candidate set (e.g., k=50 → top 5) with a cross-encoder improves precision before the context is assembled.

Embedding models must match between indexing and query time; changing the model requires full re-indexing.

Recall@k measures whether the relevant document appears in the retrieved set; precision measures how much retrieved content is relevant; track both separately from answer quality.

Metadata filtering (tenant, date, source, permission) should be applied inside the vector search, not after, to preserve recall and enforce access control.

Ground answers with citations to retrieved chunk IDs so hallucination can be detected and audited.

Evaluate retrieval independently of generation: a wrong answer with correct context is a generation failure; a wrong answer with missing context is a retrieval failure.

Build a golden dataset from real traffic with labeled expected outputs; 50-200 well-chosen cases often beat thousands of synthetic ones.

Layer evaluation: unit tests on deterministic components (parsers, validators), offline evals on model behavior, online evals on live traffic.

LLM-as-judge requires a rubric, a reference answer where possible, and calibration against human labels; report judge-human agreement.

Use pairwise comparison for preference and absolute scoring for regression tracking; fix the judge model version to keep scores comparable over time.

Track task metrics (accuracy, faithfulness, relevance), safety metrics (refusal correctness, harmful output rate), and operational metrics (latency, cost per request) separately.

Regression suites must include previously failing cases; every production incident becomes a permanent test case.

Temperature 0 reduces variance but does not guarantee determinism across runs; run repeated samples to measure output stability.

Faithfulness and groundedness checks compare claims in the answer against the retrieved context; they catch hallucination that accuracy alone can miss.

Prompt injection is a data/instruction boundary failure: treat all retrieved and user-supplied content as untrusted data, never as instructions.

Defense in depth: input filtering, instruction hierarchy in the system prompt, tool-call allowlists, output filtering, and least-privilege credentials for tools.

Never place secrets, API keys, or full authorization logic in the system prompt; the prompt is not a security boundary.

Authorization must be enforced server-side at tool execution and retrieval time, using the caller's identity, not the model's claims.

PII handling: detect and redact before sending to the model where feasible; log retention policies must cover prompts, completions, and retrieved documents.

Content filters and classifiers belong both pre-generation (blocking disallowed requests) and post-generation (blocking disallowed outputs).

Governance artifacts: model card, intended-use statement, risk assessment, human oversight plan, and an incident response runbook for harmful outputs.

Auditability requires logging the model version, prompt template version, retrieval sources, and tool calls for every consequential decision.

Latency decomposes into queueing, retrieval, time-to-first-token, and generation time; streaming improves perceived latency without reducing total time.

Set timeouts and retries with exponential backoff plus jitter; retry only idempotent operations and cap total attempts.

Rate limits are usually measured in requests per minute and tokens per minute; provision for both and queue or shed load when either is exceeded.

Cost levers in order of impact: prompt caching, output length control, model tiering/routing, retrieval depth reduction, and batching.

Cache aggressively at multiple levels: exact-match response cache, semantic cache for near-duplicate queries, and provider prompt cache for stable prefixes.

A canary or shadow deployment compares a new model or prompt against production traffic before full rollout; keep a one-flag rollback path.

Monitor token usage per request and per tenant; unbounded conversation history is a common silent cost driver.

Define SLOs on p95 latency and error rate, and alert on quality drift (eval score regression) as well as availability.