Databricks Context Engineer Associate cheat sheet

Context engineering is the discipline of selecting, shaping, storing, and serving the token payload an LLM sees; it differs from prompt engineering in that it treats retrieval, chunking, and freshness as data-engineering problems.

Databricks workspace primitives relevant to context: Unity Catalog for governed assets, Delta Lake tables for source data, Model Serving for LLM and embedding endpoints, Vector Search for indexes, and Workflows/Jobs for orchestration.

Delta Lake guarantees used by context pipelines: ACID transactions, schema enforcement and evolution, time travel for reproducibility, and Change Data Feed for incremental re-embedding.

A RAG request path in Databricks: query -> embedding endpoint -> Vector Search index query -> retrieved chunks -> context assembly -> foundation model endpoint -> response, with MLflow tracing wrapping the chain.

Compute choice rule: use serverless or job clusters for batch embedding and ETL, and provisioned throughput endpoints for latency-sensitive serving; interactive clusters are for development only.

Foundation Model APIs provide pay-per-token access to hosted models; provisioned throughput gives dedicated capacity with a fixed token-per-second rate for production workloads.

Unity Catalog three-level namespace is catalog.schema.object; context assets should be registered there so lineage, access control, and discovery apply uniformly.

MLflow 3 treats GenAI apps as logged models with traces; each retrieval and LLM call becomes a span, enabling latency and quality debugging per request.

Ingest with Auto Loader (cloudFiles) for incremental file arrival into Delta; it tracks discovered files in a checkpoint, so re-runs do not duplicate rows.

Parsing rule: use binaryFile plus a parser (ai_parse_document or a library) for PDFs and Office files; unstructured text must be extracted before chunking, and parse failures should be routed to a quarantine table.

Chunking decision rule: fixed-size token windows with 10-20% overlap are the default; semantic or structure-aware chunking is preferred when documents have headings, tables, or code blocks.

Store chunk metadata alongside text: source path, document id, page, section heading, ingestion timestamp, and content hash; the hash enables skipping unchanged chunks on re-embedding.

Deduplicate before embedding using content hashes or MinHash-style near-duplicate detection; duplicate chunks crowd out diverse context at retrieval time.

Clean text deterministically: strip boilerplate headers/footers, normalize whitespace and unicode, and remove navigation artifacts; record cleaning version so results are reproducible.

Delta Live Tables or Lakeflow pipelines with expectations (expect, expect_or_drop) enforce quality at ingestion; quarantining beats silent dropping for auditability.

Keep the parsed and chunked tables as the system of record; embeddings are derived artifacts that can be rebuilt from them at any time.

Delta Lake is the source-of-truth store for chunks and metadata; Delta Sync can maintain a Vector Search index directly from a Delta table so index refresh follows table updates.

Vector Search index types: Delta Sync indexes auto-track source table changes; Direct Vector Access indexes are written via the API and must be updated explicitly.

Similarity metric choice: cosine for normalized embeddings (most text models), L2 for unnormalized vectors; the metric must match how the embedding model was trained.

Hybrid retrieval combines dense vector similarity with keyword (BM25) scoring; fusion via reciprocal rank fusion (RRF) is a standard, tuning-light way to merge the two ranked lists.

Metadata filtering (for example by document type, date, or access group) should be applied inside the index query, not after retrieval, to avoid wasting top-k slots.

Context assembly rule: order retrieved chunks by relevance, deduplicate overlapping spans, respect a token budget, and include source identifiers so answers can cite provenance.

Reranking with a cross-encoder over the top 20-50 candidates improves precision before the final top-k is passed to the LLM.

Delta tables should store embeddings as ARRAY alongside chunk text; keeping vectors in the same table as metadata simplifies filtering and rebuilds.

Embedding endpoint calls are batched; typical batch sizes are 100-1000 inputs per request, and throughput scales with endpoint concurrency rather than single-request size.

Embedding model changes invalidate the index: vectors from different models are not comparable, so a model swap requires re-embedding the full corpus into a new index and cutover.

Store the embedding model name and version with every vector row; this makes mixed-version detection and partial rebuilds possible.

Incremental embedding pattern: use Change Data Feed or a content hash diff to compute only new or changed chunks, then upsert into the index.

Vector Search endpoints have configurable types (standard vs storage-optimized) and scale with the number of vectors; index build is asynchronous, so pipelines must poll status before querying.

Query-time embedding must use the same model, same normalization, and same instruction prefix (if the model requires one, e.g. query vs passage prefixes) as the corpus embedding.

Chunk embedding cost is roughly proportional to token count; long chunks raise cost and dilute similarity, so 200-800 token chunks are a common starting range.

Monitor embedding drift by sampling query vectors and comparing their distribution to the corpus centroid; large shifts suggest vocabulary or domain change.

Use a retrieval chain abstraction (for example LangChain or LlamaIndex integrated with Databricks) or the Databricks Agent Framework; log the chain as an MLflow model for versioning.

Agent tools should be narrow and typed; each tool call is a trace span, so tool granularity determines observability quality.

Orchestration with Databricks Workflows: separate tasks for ingestion, chunking, embedding, index sync, and evaluation; task dependencies enforce ordering.

Conversational context rule: keep a rolling window of recent turns plus a summarized older history; raw full history inflates cost and dilutes the retrieved context.

Guardrails belong at the application layer: validate retrieved context is non-empty before calling the LLM, and return a fallback answer when retrieval confidence is low.

Model Serving endpoints support AI Gateway features (rate limits, payload logging, usage tracking) that centralize cross-cutting concerns instead of reimplementing them per app.

Prompt templates should be versioned artifacts stored in Unity Catalog or MLflow, not inline strings, so changes are reviewable and rollback-able.

Set explicit token budgets per request: system prompt + retrieved context + history + reserved output tokens must fit the model context window.

Unity Catalog governs both the Delta source tables and the Vector Search index; index access honors the underlying table permissions and row/column filters.

Row filters and column masks applied to source tables propagate to Delta Sync indexes, which is the mechanism for per-user retrieval scoping.

PII handling rule: detect and redact sensitive fields before embedding; embeddings can leak information, so treat vectors as sensitive data with the same controls as text.

Lineage in Unity Catalog links source tables to derived chunk tables and indexes, supporting impact analysis when a source is corrected or deleted.

Data quality expectations at ingestion (null checks, length bounds, language detection) prevent garbage chunks from entering the index; failing rows go to a quarantine table.

Audit logging via system tables (system.access.audit) records who queried which endpoint and table, which is needed for compliance reporting.

Retention and deletion: when a source document is deleted, its chunks and vectors must be removed from both the Delta table and the index; Delta Sync handles this if the source row is deleted.

Secrets for external model APIs belong in Databricks secret scopes and are referenced in code, never hardcoded in notebooks or prompt templates.

Retrieval metrics: recall@k and precision@k measure whether the right chunks are found; MRR and nDCG account for ranking position.

Generation metrics: faithfulness (answer supported by retrieved context), answer relevance, and context precision/recall are the standard RAG triad; MLflow evaluate computes them with LLM judges.

Build a golden evaluation set of question-answer pairs with labeled supporting chunks; without labeled chunks you can only score generation, not retrieval.

LLM-as-judge scores should be validated against human labels on a sample; report agreement so judge reliability is known.

Monitor production traces for latency percentiles, token usage, retrieval hit rate, and refusal/fallback rate; alert on regressions against the evaluation baseline.

Optimization levers in order of typical impact: chunking strategy, hybrid retrieval plus reranking, metadata filtering, prompt/context ordering, then model choice.

Cache embeddings for repeated queries and cache full responses for frequent identical questions; cache hit rate is a direct cost and latency lever.

Track cost per request as embedding tokens + LLM input tokens + LLM output tokens; context assembly decisions dominate input token cost.