Context Engineering Foundations and the Databricks Platform
This domain tests whether you can translate a product requirement into a Databricks context pipeline: what evidence to retrieve, where it lives, how it is governed, and how it reaches the model at request time. The exam treats context engineering as an architecture discipline, not prompt wording alone.
The four context layers
- Instructions: role, policy, output schema, refusal rules.
- Retrieved evidence: chunks, rows, or documents selected for this request.
- Tool results: governed functions the model calls for fresh or transactional data.
- State: conversation turns, user attributes, session memory. Each layer has a different freshness and permission profile, so do not collapse them into one blob.
Platform mapping
Delta Lake tables are the durable source of truth; vector indexes and caches are derived artifacts that can be rebuilt. Unity Catalog is the permission and lineage boundary for tables, functions, models, and vector indexes. Mosaic AI Vector Search handles retrieval, Model Serving hosts the application or model, and MLflow traces capture what context was assembled. Ingestion and serving should be separate jobs: Auto Loader or Lakeflow pipelines populate bronze and silver tables, while online retrieval reads curated gold tables and indexes.
Worked example
A support assistant must answer policy questions and check order status. Policy PDFs change hourly; orders are transactional and must be fresh. Design: land PDFs with Auto Loader into a bronze Delta table, parse and chunk into silver with doc_id, version, section, and ACL tags; create a Delta Sync Vector Search index over the chunk table. Order status should not be embedded stale; expose a Unity Catalog function as a tool. At request time, retrieve top-k policy chunks, call the order tool only when needed, and assemble a prompt with policy instructions, citations, and a token budget. Log retrieved chunk IDs and tool calls in MLflow. Reasoning: hourly document change favors Delta Sync; transactional freshness favors a tool; ACL tags must travel with chunks because retrieval is not a table scan.
Traps
- Treating context as only the prompt string and ignoring retrieval, tools, and state.
- Embedding transactional data that goes stale in minutes.
- Forgetting to reserve tokens for the answer and citations.
- Assuming a vector index inherits all source-table row-level semantics without designing ACL metadata.