Scalability of Agentic AI Solutions
Domain 2 - Scalability of agentic AI solutions. This domain tests how you keep latency and cost predictable when one user request expands into dozens of model and tool calls, and how you map that load onto NVIDIA serving infrastructure.
Find the amplification factor first
Agentic workloads multiply: one task may trigger planning, retrieval, several tool calls, and a verification pass. Before adding GPUs, count model calls per task, tokens per call, and fan-out width. Reduce before you scale: route with a small model, cache planning and retrieval results, parallelize independent tool calls, cap reflection rounds, and pass references instead of full documents.
Serving and infrastructure
Serve models as NIM microservices on Kubernetes, with TensorRT-LLM-optimized engines and Triton Inference Server where multi-framework serving is needed. Enable continuous batching, prefix caching, and KV-cache-aware routing so shared system prompts and long agent histories are not recomputed. Disaggregated prefill/decode, as in NVIDIA Dynamo, separates compute-bound prefill from memory-bound decode so each pool is sized independently. Right-size models: small quantized models for routing, classification, and extraction; larger models only for reasoning steps. Use MIG partitioning for small models and reserve full GPUs for large ones.
Worked example
A research agent issues eight subqueries per user. Naively that is eight concurrent large-model calls per user, saturating decode at 50 users. Redesign: a small router model plans, eight retrievals run in parallel against a cached index, and the large model synthesizes once. Large-model calls drop from nine to one, with nine small calls; p95 latency falls and KV-cache memory shrinks.
Traps
Adding replicas before reducing call amplification; autoscaling on CPU instead of queue depth or time-to-first-token; planning from average latency instead of p95; ignoring KV-cache memory; synchronous tool calls with no timeout, retry budget, or backpressure; keeping session state in worker memory instead of an external store.