Azure Databricks Data Engineer Associate key terms

The account-level governance layer that stores metadata for tables, volumes, models, and functions, and controls access through a three-level namespace of catalog, schema, and object.

The top-level Unity Catalog container attached to a region that holds catalogs and their metadata; one metastore is assigned to each workspace in a region.

A Unity Catalog object that pairs a cloud storage path with a storage credential, letting users create external tables and volumes at that path without embedding long-lived cloud keys.

A Unity Catalog object wrapping an IAM role or service principal that grants Databricks permission to access a cloud storage bucket or container.

A table whose data files and metadata are both managed by Databricks in the metastore's default storage location; dropping it deletes the underlying data.

A table whose metadata is registered in the metastore but whose data files live at a user-specified path; dropping the table leaves the files intact.

The open table format that adds a transaction log on top of Parquet files, enabling ACID transactions, schema enforcement, time travel, and efficient upserts on data lake storage.

The _delta_log directory of JSON commits and Parquet checkpoints that records every change to a Delta table and is the source of truth for its current state.

The ability to query an earlier version of a Delta table using a version number or timestamp clause, useful for auditing, reproducing results, and recovering from bad writes.

Delta Lake's write-conflict model where transactions proceed in parallel and are validated at commit time; conflicting commits fail rather than blocking readers and writers upfront.

A SQL and DataFrame operation that matches source rows to target rows and applies conditional inserts, updates, and deletes in a single atomic Delta transaction.

A declarative Lakeflow pipeline feature that applies change data capture feeds to a target table, handling ordering, deduplication, and SCD type 1 or type 2 history automatically.

A Structured Streaming source that incrementally discovers new files in cloud storage, tracks them in a checkpoint, and infers or evolves schema as data arrives.

The ability to automatically add newly observed columns to a table or stream schema as source data changes, controlled by options such as addNewColumns or mergeSchema.

Delta Lake's rejection of writes whose data types or columns do not match the target table schema, protecting table quality unless the write explicitly opts into evolution.

A Lakeflow pipeline materialized table that is continuously refreshed from a streaming query, with data quality expectations and dependency tracking handled by the pipeline.

A Lakeflow pipeline object whose results are recomputed from its defining query on each refresh, typically used for aggregations and joins over batch or streaming sources.

A declarative data pipeline defined in code whose tables and views are built and refreshed by the Databricks runtime, with automatic dependency ordering and incremental processing.

Data quality constraints declared on a pipeline dataset that record or drop invalid rows, or fail the update, depending on the configured action.

A scheduled or triggered workflow that runs one or more tasks, such as notebooks, SQL, pipelines, or Python scripts, with dependencies, retries, and notifications.

A directed relationship in a job that makes one task run only after another succeeds, forming a DAG that determines execution order and parallelism.

A cluster created for a single job run and terminated afterward, isolating workloads and reducing cost compared with an always-running all-purpose cluster.

A shared interactive cluster used for notebooks and exploration, kept running across sessions and typically more expensive per unit of compute than job clusters.

A rule set that constrains which cluster attributes users may configure, such as node types, autoscaling limits, and tags, enforcing cost and governance standards.

A job re-run that executes only the failed or skipped tasks plus their downstream dependencies, reusing successful task results from the original run.

A project structure that declares jobs, pipelines, notebooks, and their settings in YAML, enabling version control, parameterization per target, and repeatable deployment.

The CLI command that validates and applies a bundle definition to a workspace target, creating or updating the declared jobs, pipelines, and files.

A non-human identity used by automated jobs and CI/CD pipelines to authenticate to Databricks and cloud storage without tying access to an individual user.

A revocable string credential that authenticates API and CLI calls as a user, commonly used in local development and simple automation.

A Unity Catalog function that returns a filtered result based on the caller's identity or group membership, used to implement row-level and column-level security.

A Unity Catalog table property that applies a function to restrict which rows a given user or group can see when querying the table.

A Unity Catalog table property that applies a function to replace or obscure column values for users who lack the privilege to see the raw data.

An open protocol for sharing Delta tables and other data with external recipients across organizations without copying the data or requiring the same platform.

A Delta command that compacts small files into larger ones and, when Z-ordering is requested, co-locates related data to improve data-skipping on filtered reads.

A multi-dimensional clustering technique applied during OPTIMIZE that sorts data by several columns so files contain tighter value ranges, improving file pruning.

A Delta layout where clustering keys are declared on the table and data is incrementally reorganized without rewriting the whole table, replacing partitioning and Z-order in many cases.

The use of per-file minimum and maximum statistics in the Delta log to skip reading files that cannot contain rows matching a query's filter predicates.

A Delta command that deletes data files no longer referenced by the transaction log and older than a retention threshold, reclaiming storage while limiting time travel.

The web interface exposing stages, tasks, shuffles, and executor metrics for a running or completed Spark application, used to diagnose skew and spills.

A Spark optimizer feature that re-plans a query at runtime using actual statistics, coalescing shuffle partitions, switching join strategies, and splitting skewed partitions.

An uneven distribution of data across partitions that leaves a few tasks processing most of the rows, causing long stragglers and underused executors.