Design and build generative AI applications
Start with the interface and the data path
A Databricks generative AI application is an endpoint-consuming client plus governed data and models. Know the building blocks: Foundation Model APIs (pay-per-token or provisioned throughput), custom Model Serving endpoints for fine-tuned or packaged models, Vector Search endpoints, and AI Functions such as ai_query for SQL inference. Applications call endpoints through OpenAI-compatible clients, LangChain, or Databricks SDKs. Prompts are templates with system instructions, few-shot examples, and retrieved context.
Design choices
Decide synchronous request/response versus batch inference over a Delta table, and single-turn versus agentic tool use. For agents, define tools, state, and a stopping condition. Separate retrieval from generation so each can be evaluated and tuned independently. Keep models, indexes, and tables in Unity Catalog so permissions and lineage follow the application.
Traps
- A serving endpoint is not a cluster; you do not attach a notebook to it.
- Pay-per-token Foundation Model APIs are good for prototyping but have rate limits and no reserved capacity. Provisioned throughput reserves capacity for predictable latency and cost at volume.
- Endpoint names are not model names; verify the exact endpoint you call.
- Do not hard-code tokens; use secrets or service credentials.
- Tool-using agents fail silently when tool schemas are ambiguous.
Rule of thumb
Prototype with pay-per-token APIs; move to provisioned throughput when traffic is steady or latency SLOs matter. Example: a support assistant at 50 requests/minute with a 2-second SLO should use a provisioned-throughput endpoint and cache frequent answers, not a pay-per-token endpoint.