NVIDIA NCP-AAI cheat sheet

Agent loop = perceive → plan → act (tool call) → observe → reflect; a ReAct agent interleaves reasoning traces with tool calls, while a plan-and-execute agent commits to a full plan first and replans only on failure.

Orchestration topologies: supervisor (a router agent delegates to specialists and aggregates), hierarchical (supervisors of supervisors), peer-to-peer/blackboard (agents read-write a shared state), and pipeline (fixed sequential handoff). Choose supervisor when subtasks are heterogeneous; choose pipeline when order is fixed and latency matters.

Tool/function calling contract: name, description, JSON-schema parameters, and return payload all consume context. Ambiguous descriptions are the top cause of wrong tool selection - write descriptions as selection criteria, not documentation.

Multi-agent failure modes to design against: infinite delegation loops, context overflow from relaying full transcripts, conflicting writes to shared state, and error cascades where one agent's bad output is trusted downstream. Mitigate with max-iteration caps, summarized handoffs, and explicit validation steps.

Handoff payloads should carry a compact structured summary (goal, constraints, findings, open questions, artifacts by reference) rather than raw conversation history; passing full transcripts scales context cost with hop count.

Parallelism is safe only for independent subtasks with no shared mutable state; fan-out/fan-in requires a deterministic merge step (dedupe, conflict resolution, or a judge agent) before results reach the user.

Memory tiers: in-context scratchpad (ephemeral), external vector store for semantic recall, structured database for durable facts, and episodic logs for replay. Retrieval quality, not model size, usually bounds long-horizon agent accuracy.

Guardrail placement: input validation before the planner, tool-argument validation before execution, and output verification before returning to the user or to another agent; each boundary is an independent trust boundary.

Cost per task ≈ (input tokens + output tokens) × price, summed over every hop; multi-agent designs multiply token spend by hop count, so track tokens-per-completed-task as a first-class SLO.

Prompt caching (prefix caching) rewards stable, long system prompts and tool schemas placed first; volatile content (user turn, retrieved chunks) belongs at the end of the prompt to maximize cache hits.

Latency decomposes into queueing, prefill (prompt processing), decode (token generation), and tool round-trips. Streaming hides decode latency from users but not tool latency; parallel tool calls cut round-trip serialization.

Throughput scaling levers: batch/queue requests, route easy queries to smaller models (model cascade), cache deterministic tool results, and cap agent iterations. Retries must use exponential backoff with jitter to avoid synchronized thundering herds.

Stateless workers plus an external session/state store allow horizontal scaling; sticky sessions break autoscaling and should be avoided for agent services.

NVIDIA NIM microservices expose models behind an OpenAI-compatible API and scale as containers on Kubernetes; GPU memory, not CPU, is the binding constraint, so size replicas by concurrent KV-cache demand.

Observability for scale: per-hop traces (OpenTelemetry-style spans), token and cost counters, tool error rates, p95/p99 latency, and loop-detection counters; without per-hop tracing, cost regressions are invisible.

Rate limits and quotas apply per model, per API key, and per GPU pool; design admission control and graceful degradation (shorter context, smaller model, cached answer) rather than hard failure.

Least-privilege tooling: each agent gets only the scopes it needs, credentials are short-lived and scoped per task, and irreversible actions (payments, deletions, external sends) require human approval or a two-step confirm.

Defense in depth against prompt injection: treat all retrieved and tool-returned text as untrusted data, separate instructions from data in the prompt structure, and never let tool output expand an agent's permissions.

Governance artifacts: an AI system inventory, risk classification, model and prompt version registry, data provenance records, evaluation results, and an incident log - each mapped to an accountable owner.

Human oversight models: human-in-the-loop (approve each action), human-on-the-loop (monitor and intervene), and human-out-of-the-loop (post-hoc audit); the choice follows the reversibility and blast radius of the action.

Transparency obligations: disclose that the user is interacting with an AI, log the reasoning trace and tool calls for auditability, and provide an explanation path for adverse decisions affecting people.

Bias and fairness controls: define protected attributes, measure outcome disparities across groups on a fixed eval set, and document mitigations; fairness metrics trade off against each other and the choice must be recorded.

Data governance: minimize personal data in prompts, redact before logging, honor retention and deletion requests across vector stores and caches, and keep training/eval data lineage separate from production traces.

Evaluation and red-teaming: maintain a regression suite of adversarial prompts (jailbreaks, injection, data exfiltration), run it on every prompt or model change, and gate deployment on pass rates rather than manual spot checks.

NVIDIA stack mapping: NIM for model inference microservices, NeMo for customization (fine-tuning, guardrails, evaluation), and agent orchestration frameworks running against the OpenAI-compatible NIM endpoints.

RAG pipeline stages: chunk (respect semantic boundaries), embed, index (vector plus metadata filters), retrieve (hybrid dense + keyword), rerank, then generate with citations; reranking typically improves precision more than swapping embedding models.

Fine-tuning vs. RAG: use RAG for changing factual knowledge and citations; use fine-tuning (LoRA/adapters) for stable style, format, or tool-calling behavior. Combining both is common; fine-tuning does not reliably add new facts.

Evaluation layers: component evals (retrieval recall@k, tool-selection accuracy), trajectory evals (did the agent take a valid path), and end-to-end task success with an LLM-as-judge rubric validated against human labels.

Deployment pattern: containerize the agent service and NIMs, deploy on Kubernetes with health/readiness probes, blue-green or canary rollout keyed on task-success and cost metrics, and keep prompts and tool schemas in version control.

Structured output enforcement: constrain generation with JSON schema or grammar-guided decoding, then validate and repair; free-form parsing of model output is a common production failure.

Caching layers: exact-match response cache, semantic cache for near-duplicate queries (with a similarity threshold), and tool-result caches with TTLs; each needs an invalidation policy tied to data freshness.

Rollback readiness: version prompts, tool definitions, and model endpoints together as one deployable unit so a regression can be reverted atomically; log the version tuple with every trace.

Agent loop = perceive → plan → act (tool call) → observe → reflect; a ReAct agent interleaves reasoning traces with tool calls, while a plan-and-execute agent commits to a full plan first and replans only on failure.

Orchestration topologies: supervisor (a router agent delegates to specialists and aggregates), hierarchical (supervisors of supervisors), peer-to-peer/blackboard (agents read-write a shared state), and pipeline (fixed sequential handoff). Choose supervisor when subtasks are heterogeneous; choose pipeline when order is fixed and latency matters.

Tool/function calling contract: name, description, JSON-schema parameters, and return payload all consume context. Ambiguous descriptions are the top cause of wrong tool selection - write descriptions as selection criteria, not documentation.

Multi-agent failure modes to design against: infinite delegation loops, context overflow from relaying full transcripts, conflicting writes to shared state, and error cascades where one agent's bad output is trusted downstream. Mitigate with max-iteration caps, summarized handoffs, and explicit validation steps.

Handoff payloads should carry a compact structured summary (goal, constraints, findings, open questions, artifacts by reference) rather than raw conversation history; passing full transcripts scales context cost with hop count.

Parallelism is safe only for independent subtasks with no shared mutable state; fan-out/fan-in requires a deterministic merge step (dedupe, conflict resolution, or a judge agent) before results reach the user.

Memory tiers: in-context scratchpad (ephemeral), external vector store for semantic recall, structured database for durable facts, and episodic logs for replay. Retrieval quality, not model size, usually bounds long-horizon agent accuracy.

Guardrail placement: input validation before the planner, tool-argument validation before execution, and output verification before returning to the user or to another agent; each boundary is an independent trust boundary.

Cost per task ≈ (input tokens + output tokens) × price, summed over every hop; multi-agent designs multiply token spend by hop count, so track tokens-per-completed-task as a first-class SLO.

Prompt caching (prefix caching) rewards stable, long system prompts and tool schemas placed first; volatile content (user turn, retrieved chunks) belongs at the end of the prompt to maximize cache hits.

Latency decomposes into queueing, prefill (prompt processing), decode (token generation), and tool round-trips. Streaming hides decode latency from users but not tool latency; parallel tool calls cut round-trip serialization.

Throughput scaling levers: batch/queue requests, route easy queries to smaller models (model cascade), cache deterministic tool results, and cap agent iterations. Retries must use exponential backoff with jitter to avoid synchronized thundering herds.

Stateless workers plus an external session/state store allow horizontal scaling; sticky sessions break autoscaling and should be avoided for agent services.

NVIDIA NIM microservices expose models behind an OpenAI-compatible API and scale as containers on Kubernetes; GPU memory, not CPU, is the binding constraint, so size replicas by concurrent KV-cache demand.

Observability for scale: per-hop traces (OpenTelemetry-style spans), token and cost counters, tool error rates, p95/p99 latency, and loop-detection counters; without per-hop tracing, cost regressions are invisible.

Rate limits and quotas apply per model, per API key, and per GPU pool; design admission control and graceful degradation (shorter context, smaller model, cached answer) rather than hard failure.

Least-privilege tooling: each agent gets only the scopes it needs, credentials are short-lived and scoped per task, and irreversible actions (payments, deletions, external sends) require human approval or a two-step confirm.

Defense in depth against prompt injection: treat all retrieved and tool-returned text as untrusted data, separate instructions from data in the prompt structure, and never let tool output expand an agent's permissions.

Governance artifacts: an AI system inventory, risk classification, model and prompt version registry, data provenance records, evaluation results, and an incident log - each mapped to an accountable owner.

Human oversight models: human-in-the-loop (approve each action), human-on-the-loop (monitor and intervene), and human-out-of-the-loop (post-hoc audit); the choice follows the reversibility and blast radius of the action.

Transparency obligations: disclose that the user is interacting with an AI, log the reasoning trace and tool calls for auditability, and provide an explanation path for adverse decisions affecting people.

Bias and fairness controls: define protected attributes, measure outcome disparities across groups on a fixed eval set, and document mitigations; fairness metrics trade off against each other and the choice must be recorded.

Data governance: minimize personal data in prompts, redact before logging, honor retention and deletion requests across vector stores and caches, and keep training/eval data lineage separate from production traces.

Evaluation and red-teaming: maintain a regression suite of adversarial prompts (jailbreaks, injection, data exfiltration), run it on every prompt or model change, and gate deployment on pass rates rather than manual spot checks.

NVIDIA stack mapping: NIM for model inference microservices, NeMo for customization (fine-tuning, guardrails, evaluation), and agent orchestration frameworks running against the OpenAI-compatible NIM endpoints.

RAG pipeline stages: chunk (respect semantic boundaries), embed, index (vector plus metadata filters), retrieve (hybrid dense + keyword), rerank, then generate with citations; reranking typically improves precision more than swapping embedding models.

Fine-tuning vs. RAG: use RAG for changing factual knowledge and citations; use fine-tuning (LoRA/adapters) for stable style, format, or tool-calling behavior. Combining both is common; fine-tuning does not reliably add new facts.

Evaluation layers: component evals (retrieval recall@k, tool-selection accuracy), trajectory evals (did the agent take a valid path), and end-to-end task success with an LLM-as-judge rubric validated against human labels.

Deployment pattern: containerize the agent service and NIMs, deploy on Kubernetes with health/readiness probes, blue-green or canary rollout keyed on task-success and cost metrics, and keep prompts and tool schemas in version control.

Structured output enforcement: constrain generation with JSON schema or grammar-guided decoding, then validate and repair; free-form parsing of model output is a common production failure.

Caching layers: exact-match response cache, semantic cache for near-duplicate queries (with a similarity threshold), and tool-result caches with TTLs; each needs an invalidation policy tied to data freshness.

Rollback readiness: version prompts, tool definitions, and model endpoints together as one deployable unit so a regression can be reverted atomically; log the version tuple with every trace.