AB-620 cheat sheet

Requirement triage rule: classify each stated need as deterministic logic, classification, extraction, or generative - deterministic logic belongs in code, not in a model prompt.

Latency budget formula: end-to-end p95 = model p95 + retrieval p95 + network overhead; a design fails the SLA if any single component's p95 exceeds the total budget.

Context window sizing: tokens needed = system prompt + retrieved chunks + conversation history + max output; if the sum exceeds the window, reduce chunk count before reducing output limit.

Model selection order: capability threshold first, then latency, then cost per 1K tokens, then regional availability; a cheaper model below the capability threshold is not a valid option.

RAG vs fine-tuning rule: choose retrieval when facts change frequently or must be cited; choose fine-tuning when the gap is tone, format, or task-specific behavior on stable knowledge.

Chunking defaults: 200-800 tokens per chunk with 10-20% overlap; overlap exists to prevent answer loss at chunk boundaries, not to increase recall of unrelated content.

Cost estimate formula: monthly cost = (input tokens x input price) + (output tokens x output price), summed over expected request volume, including retries and evaluation traffic.

Human-in-the-loop trigger design: require review when model confidence is below threshold, when the action is irreversible, or when the request matches a high-risk intent class.

Tool schema rule: every tool parameter needs a type, a description, and a required flag; ambiguous descriptions are the most common cause of malformed tool calls.

Agent loop pattern: observe, plan, call tool, evaluate result, repeat - cap iterations explicitly because unbounded loops are the primary driver of runaway token spend.

Orchestration choice: use a fixed workflow when steps are known in advance; use an autonomous agent only when the next step depends on runtime results.

Grounding rule: an agent must cite the retrieved record or tool response that supports a factual claim; ungrounded claims are treated as hallucinations regardless of plausibility.

Handoff design: pass a structured summary object between agents (intent, constraints, collected facts) rather than the raw transcript, to control token growth.

Idempotency requirement: any tool that creates or mutates state must accept an idempotency key so retries after a timeout do not duplicate side effects.

Fallback chain: primary model, then secondary model, then deterministic scripted response; each hop must log the reason for the fallback for later tuning.

Prompt structure order: role and constraints, then task, then context, then output format; output format instructions placed last are followed more reliably.

Retrieval pipeline stages in order: ingest, chunk, embed, index, query-embed, similarity search, optional rerank, then prompt assembly.

Hybrid search rule: combine vector similarity with keyword search and merge with reciprocal rank fusion when queries contain exact identifiers, codes, or product names.

Embedding consistency requirement: query and document vectors must come from the same model and version; re-embed the corpus whenever the embedding model changes.

API integration pattern: wrap model calls in a service layer that handles auth, retries with exponential backoff, timeouts, and response validation before downstream use.

Freshness design: index updates should be event-driven from the source system; a nightly batch alone creates a staleness window equal to the batch interval.

Chunk metadata minimum: source id, document title, section path, last-modified timestamp, and access-control tags; access tags are required to filter results per user.

Rate limit handling: read the retry-after header when present, otherwise back off exponentially with jitter; fixed-interval retries amplify throttling.

Data contract rule: define the schema, nullability, and expected value ranges for every field passed to the model, and reject payloads that fail validation.

Evaluation set design: hold out a labeled set covering common, edge, and adversarial cases; report per-category scores because an aggregate score hides regressions.

Core metrics: groundedness (claims supported by context), relevance (answer addresses the question), and retrieval recall (relevant chunk present in top-k).

Regression gate rule: a prompt or model change ships only if it does not decrease any per-category metric below its defined floor.

Online monitoring signals: latency percentiles, error and refusal rate, token usage per request, tool-call failure rate, and user thumbs-down rate.

Drift detection: track input topic distribution and output length distribution over time; a shift beyond the baseline band triggers re-evaluation.

Optimization order: fix retrieval quality first, then prompt clarity, then model choice, then caching - retrieval errors dominate most quality failures.

Caching rule: cache identical normalized prompts with a TTL tied to data freshness; disable caching for personalized or permission-filtered responses.

A/B comparison rule: change one variable at a time (prompt, model, temperature, retrieval depth) so the observed delta is attributable.

Prompt injection defense: treat all retrieved and user-supplied content as untrusted data, never as instructions; enforce tool permissions outside the model.

Least privilege rule: each agent identity gets only the scopes its tools require; the model must not hold credentials that allow broader access than the workflow needs.

PII handling: detect and redact sensitive fields before sending to the model, and log only identifiers or hashes rather than raw content.

Deployment gate: require evaluation pass, security review, and a rollback plan before promotion; canary release to a small traffic slice first.

Audit requirement: log prompt version, model version, tool calls, and decision outcome for every request so any output can be reconstructed.

Content safety layering: apply input filtering, model-side safety settings, and output filtering; no single layer is sufficient on its own.

Data residency rule: route requests and store logs only in regions permitted by policy; verify the deployment's region configuration matches the data classification.

Responsible AI disclosure: users must be told they are interacting with an AI system, and high-impact decisions must retain a documented human review path.