Building and Orchestrating Context-Aware Applications
This domain tests how retrieval becomes a working application: routing, tool calling, state, deployment, and failure handling. The exam focuses on deterministic control around model reasoning, governed tools, and observable execution.
Orchestration rules
- Separate deterministic retrieval from model reasoning. The application should decide when retrieval is required, which index or filter to use, and what to do when evidence is missing.
- Register tools as Unity Catalog functions or governed endpoints. Give the model a narrow schema and grant only the privileges the tool needs.
- Set execution budgets: maximum tool calls, per-call timeouts, retry with backoff, and a fallback response when a dependency fails.
- Deploy through Model Serving and capture traces in MLflow so retrieval inputs, tool arguments, and outputs are inspectable.
State and guardrails
- Persist conversation state outside the prompt; summarize long threads instead of replaying every turn.
- Validate tool arguments server-side and enforce authorization in the tool, not in the prompt.
- Return structured errors so the orchestrator can decide whether to retry, ask for clarification, or answer partially.
Worked example
A support agent has two tools: search_docs for policy evidence and get_order_status for live orders. A router first classifies the request. Policy questions call search_docs with a product filter; order questions call get_order_status with an order_id that must belong to the authenticated user. If get_order_status times out after two retries, the agent returns the policy portion with a clear notice that order status is unavailable, rather than inventing a status. The trace records the router decision, tool arguments, latency, and retrieved chunk IDs. Reasoning: authorization belongs in the tool; bounded retries prevent loops; partial answers are safer than hallucinated transactional data.
Traps
- Letting the model loop indefinitely across tools without call limits.
- Putting authentication or permission checks in the prompt instead of server-side code.
- Omitting timeouts and fallback behavior for slow dependencies.
- Deploying without tracing, making production failures undiagnosable.
- Hardcoding secrets in prompts or notebook code instead of using governed secrets.