Databricks Context Engineer Associate key terms

The practice of selecting, shaping, storing, and delivering the right information to a model at inference time so its output is grounded, current, and relevant rather than relying on training data alone.

The maximum number of tokens a model can accept in a single request, covering system prompt, retrieved passages, conversation history, and the user question together. Exceeding it forces truncation or summarization.

The atomic unit of text a model reads and bills for, roughly a word fragment. Token counts drive context-window limits, embedding cost, and chunk-size decisions in a pipeline.

Databricks' centralized governance layer for data and AI assets. It holds catalogs, schemas, tables, volumes, models, functions, and vector indexes, and enforces permissions and lineage across all of them.

The open table format underpinning most Databricks storage. It adds a transaction log for ACID guarantees, schema enforcement, time travel, and change data feed on top of Parquet files.

A layered data design with bronze (raw ingested), silver (cleaned and conformed), and gold (business-level, retrieval-ready) tables. Each layer improves quality and narrows what downstream context consumes.

A Databricks ingestion utility that incrementally detects and processes new files landing in cloud storage. It tracks discovered files in a checkpoint, supports schema inference and evolution, and scales to large volumes.

Spark's incremental processing engine where a query runs continuously over an unbounded table. In Databricks it powers near-real-time pipelines that keep context stores fresh as source data arrives.

A Delta Lake feature that records row-level inserts, updates, and deletes so downstream consumers can read only what changed. Useful for incrementally refreshing embeddings instead of re-embedding whole tables.

The ability of a pipeline to absorb new or changed columns without breaking. Relevant when source documents gain metadata fields that must flow into chunk tables and vector index filters.

A Delta Lake behavior that rejects writes whose data does not match the table's declared schema, protecting downstream retrieval quality from malformed or unexpected columns.

Splitting source documents into smaller passages before embedding. Chunk size and overlap trade retrieval precision against lost surrounding context, and often need tuning per document type.

A chunking strategy that tries large natural boundaries first (sections, paragraphs, sentences) and falls back to smaller ones, preserving semantic units better than fixed-length character cuts.

Grouping text by meaning rather than length, for example by detecting topic shifts through embedding similarity between adjacent sentences. Often improves answer grounding at higher compute cost.

A dense numeric vector representing text meaning, produced by a model so that semantically similar passages land near each other in vector space. Stored alongside chunk text for similarity search.

The model that converts text into vectors. Choice affects dimensionality, language coverage, max input length, cost, and retrieval quality; changing it requires re-embedding the whole corpus.

A measure of the angle between two vectors, ranging from -1 to 1, used to rank passages against a query. Ignores magnitude, so it suits normalized embeddings and is the common default metric.

A Databricks-managed index over embeddings stored in a Delta table, supporting similarity queries with metadata filters. It syncs from the source table and exposes results through an SDK or SQL.

Combining vector similarity with keyword or BM25 matching and merging the ranked results. Recovers exact identifiers and rare terms that pure embedding search often misses.

A rank-merging method that scores each document by the sum of 1/(k + rank) across result lists. It blends lexical and vector rankings without needing comparable raw scores.

A second-stage scoring pass, usually a cross-encoder, that reorders an initial candidate set by fine-grained query-document relevance. Improves precision at the cost of extra latency.

An architecture that retrieves relevant passages at query time and inserts them into the prompt so the model answers from supplied evidence instead of memory alone, reducing hallucination and staleness.

Tying a model's answer to specific retrieved evidence, ideally with citations, so claims can be traced and verified. A core quality goal of any context pipeline.

Restricting vector search to records matching attribute conditions such as tenant, date, or document type. Enforces scoping and access boundaries before similarity ranking happens.

Hosted endpoints giving pay-per-token access to curated foundation models for chat and embeddings, without managing infrastructure. Useful for prototyping and for workloads that need no custom weights.

A Databricks-hosted REST API exposing a model, agent, or composed chain for real-time inference. Handles scaling, versioning, and authentication for application traffic.

An open-source framework for composing LLM applications from components such as loaders, retrievers, prompt templates, and chains. Integrates with Databricks models, vector stores, and tracing.

An LLM-driven program that decides which tools or retrieval steps to invoke and iterates toward a goal. Requires guardrails on tool access, step limits, and output validation.

A model capability to emit structured requests for external functions, such as a search or SQL query, whose results are fed back into the conversation. The basis for agentic retrieval.

Instrumentation that records each step of an LLM or agent request as a span, capturing inputs, outputs, latency, and token usage. Essential for debugging and monitoring retrieval pipelines.

A curated set of questions paired with expected answers or relevant documents, used to score retrieval and generation quality repeatably as prompts, chunking, or models change.

The degree to which every claim in a generated answer is supported by the retrieved context. A high score means little unsupported invention, independent of whether the answer is complete.

A retrieval metric measuring how much of the information needed for the reference answer appears in the retrieved passages. Low recall usually signals chunking, embedding, or top-k problems.

Access control that restricts which rows a user can see, enforced through Unity Catalog row filters or dynamic views. Critical so retrieval never returns another tenant's or region's content.

A recorded graph of how data moves from source tables through transformations to downstream tables, indexes, and models. Supports impact analysis and audit when context sources change.

A declared rule about acceptable data, such as non-null or range constraints, checked during pipeline runs. Failures can warn, drop rows, or halt, protecting retrieval from bad inputs.

An attack where untrusted content in retrieved documents or user input steers the model to ignore instructions or leak data. Mitigated by delimiting context, least-privilege tools, and output checks.

A Unity Catalog-governed routine, including Python UDFs, that can be registered, permissioned, and invoked in SQL or agents. A controlled way to expose logic to models and pipelines.