Databricks Data Engineer key terms

Open-source storage layer that adds a transaction log to Parquet files in cloud storage, providing ACID transactions, schema enforcement, time travel and efficient upserts on data lake tables.

A table whose data files and _delta_log transaction log live in cloud storage; the log records each commit so readers always see a consistent snapshot of the table.

Querying an earlier version of a Delta table by version number or timestamp, useful for auditing, reproducing past reports and recovering data after an incorrect write.

Interactive compute that a user starts and keeps running for notebooks and ad hoc analysis; it can be shared by several users but costs more per DBU than job compute.

Compute resource tuned for SQL and BI workloads, offered in serverless or classic sizes, used by dashboards, the SQL editor and clients connecting over JDBC or ODBC.

Account-level governance layer that provides a shared metastore, three-level namespaces, centralized privileges, lineage and audit records across workspaces and clouds.

Top-level Unity Catalog container that holds catalogs and is assigned to workspaces; it defines the boundary inside which principals, storage credentials and grants are managed.

Naming pattern catalog.schema.table that uniquely identifies Unity Catalog objects and lets queries resolve tables unambiguously regardless of the workspace a user is working in.

Unity Catalog object that pairs a cloud storage path with a storage credential, letting authorized users read and write external data without embedding secrets in code.

Unity Catalog object holding a cloud IAM role or managed identity that Databricks assumes to reach storage; external locations and external tables reference it.

Permission such as USE CATALOG, SELECT or MODIFY granted on a securable object to a user, group or service principal; privileges are inherited from catalog down to table.

Unity Catalog functions attached to a table that restrict which rows a caller sees or transform sensitive column values at query time, based on the caller's identity or group.

Non-human identity used by automated jobs, pipelines and CI/CD tooling; it authenticates with OAuth client credentials or a secret and is granted privileges like a user.

Structured Streaming source, configured as cloudFiles, that incrementally discovers and ingests newly arriving files from cloud storage, tracking processed files so each file is read once.

SQL command that loads files from a storage location into a Delta table idempotently, recording which files were already loaded and skipping them on subsequent runs.

Directory where a Structured Streaming query stores its progress, file offsets and state, allowing the query to resume from the correct position after a restart.

Ingestion setting that lets a pipeline add newly appearing columns to the target automatically, or capture unexpected data in a rescued column instead of failing the load.

Layered data design of bronze (raw ingested), silver (cleaned and conformed) and gold (business aggregates) tables, where each layer improves quality for downstream consumers.

Delta SQL statement that inserts, updates or deletes target rows from a source based on a match condition, enabling idempotent upserts and slowly changing dimension loads.

Declarative pipeline command that applies a change data capture feed of inserts, updates and deletes to a target streaming table, using a sequencing column to resolve out-of-order records.

Declarative pipeline object that processes each incoming record once and appends results incrementally, making it suitable for continuous ingestion and change data capture targets.

Declarative pipeline object whose query result is recomputed on each refresh; used for aggregations and joins where incremental processing is not feasible.

Framework in which you declare streaming tables, materialized views and expectations in code; the engine builds the dependency graph, manages compute and refreshes results incrementally.

Data quality constraints declared inside a declarative pipeline that record metrics and can warn, drop affected rows, or fail the update when records violate the rule.

Physically splitting table data into directories by column value, typically a date, so queries that filter on that column avoid reading unrelated data.

Delta feature that replaces fixed partitions and Z-order by selecting and maintaining clustering keys automatically as data is written and later optimized.

Delta command that compacts many small files into fewer larger ones and can apply clustering, improving read throughput and the effectiveness of file skipping.

Delta command that deletes data files no longer referenced by the transaction log and older than the retention threshold, reclaiming storage but removing older time travel history.

Spark optimization that re-plans a query at runtime using actual statistics, coalescing small shuffle partitions, splitting skewed partitions and switching join strategies.

Uneven distribution of join or grouping keys so that a few tasks process most of the data, creating long stragglers that dominate a stage's duration.

Web interface for a running or finished Spark application showing stages, tasks, shuffle, spill and executor metrics used to diagnose slow or failing jobs.

Account-level Unity Catalog tables such as system.billing.usage and system.access.audit that expose cost, audit and operational data for querying and monitoring.

Orchestration service for scheduling multi-task workflows that run notebooks, pipelines, SQL and other tasks with dependencies, triggers, retries and notifications.

Ephemeral compute created for a single Lakeflow job run and terminated when it ends; cheaper and isolated, and the recommended choice for scheduled production workloads.

Rule that starts a job run: a cron schedule, arrival of files in a location, continuous processing, or a manual or API request.

Rerunning only the failed and skipped tasks of a job run while reusing the results of tasks that already succeeded, saving time and compute cost.

YAML-defined package of jobs, pipelines, notebooks and permissions that can be validated and deployed with the CLI, giving version-controlled and repeatable deployments.

Command-line tool that authenticates to a workspace and runs bundle validate, deploy and run commands, typically invoked from CI/CD pipeline scripts.

Workspace feature that clones a remote Git repository so notebooks and files are version controlled and developers can work on branches before merging changes.

Deploying the same bundle artifact from a development workspace to staging and production, using target-specific variables, permissions and service principals.