Databricks Data Engineer Professional key terms

Incremental file ingestion source that discovers new files in cloud storage and records which ones it has already processed in a checkpoint, so each file is read once as it arrives.

SQL command that loads files from a storage location into a Delta table, remembering which files were already loaded so a rerun does not duplicate rows. Best suited to scheduled batch loads.

Directory where Spark writes stream progress, processed offsets, and state, so a restarted query continues where it stopped instead of reprocessing data or losing its place.

Streaming trigger that processes all data currently available and then stops, so a pipeline can run on a schedule at batch-like cost rather than running continuously.

Bound on how late event-time data may arrive before the engine stops waiting for it. It lets streaming aggregations discard old state and decide when to emit results.

Pipeline dataset that reads an append-only source and is refreshed incrementally, so each input row is processed once instead of the whole history being recomputed on every update.

Declarative pipeline statement that merges an ordered change feed into a target table, sequencing out-of-order events and optionally keeping full slowly changing dimension history.

Table format that stores data as Parquet files plus a transaction log, adding ACID transactions, schema enforcement, time travel, and efficient updates on top of ordinary cloud storage.

JSON log in a Delta table's _delta_log folder that records each commit; readers replay it to learn which data files make up a consistent version of the table.

Reading a Delta table as it existed at an earlier version number or timestamp, which is useful for auditing, reproducing a report, or undoing a bad write.

SQL statement that matches source rows against target rows and applies insert, update, or delete actions, allowing upserts and change-data-capture loads without rewriting the whole table.

Approach in which a stream of insert, update, and delete events from a source system is applied to a target table so the two stay consistent, usually through MERGE or APPLY CHANGES.

Allowing a table or stream to accept added or changed columns without rewriting the pipeline, controlled by options such as mergeSchema, addNewColumns, or overwriteSchema.

Data-quality rule declared on a pipeline dataset together with what to do when it fails (warn, drop the row, or fail the update), with results written to the pipeline event log.

Open protocol for giving external recipients read access to selected tables without copying files; recipients use any compatible client while the provider keeps control of the data.

Unity Catalog capability to query tables in external systems such as PostgreSQL, MySQL, Snowflake, or BigQuery through a connection, reading the data in place instead of ingesting it.

Read-only tables in the system catalog holding account-level operational data such as billable usage, audit records, and job or query history, queryable with ordinary SQL.

Append-only record of a pipeline's updates, data-quality results, and errors, stored as a Delta table and queried to monitor pipeline health and troubleshoot failures.

Callback interface registered on a Spark session that receives events as a stream starts, progresses, or terminates, used to emit custom metrics and drive alerts.

Delta table layout that clusters data on chosen keys and lets those keys change without a full rewrite; OPTIMIZE incrementally reorganises files so data skipping keeps working.

Delta optimisation that sorts data so rows with similar values in the chosen columns land in the same files, letting queries skip files that cannot match their filters.

Managed feature that observes workload and automatically runs OPTIMIZE, VACUUM, and ANALYZE on Unity Catalog managed tables, replacing hand-written maintenance jobs.

Operation that deletes Delta data files no longer referenced by the log and older than a retention threshold. It lowers storage cost but shortens how far time travel can reach.

Databricks' vectorised C++ query engine that replaces parts of Spark's execution engine, speeding up SQL and DataFrame work such as scans, joins, and aggregations.

Runtime re-planning that uses actual shuffle statistics to merge small partitions, switch to broadcast joins, and split skewed partitions, improving joins without manual tuning.

Join in which a small table is copied to every executor so the large table need not be shuffled across the network; effective when the small side fits in memory.

Account-level governance layer providing a catalog.schema.table namespace plus access control, lineage, audit logging, and discovery that apply consistently across workspaces.

Top-level Unity Catalog container that stores catalogs, storage credentials, and external locations and is attached to the workspaces in a region.

Automatically captured record of which tables and columns each query read and wrote, used for impact analysis, auditing, and tracing bad data back to its source.

Unity Catalog object pairing a cloud storage path with a storage credential, letting users read and write those files under governed permissions instead of raw secrets.

Unity Catalog object holding a cloud IAM role or managed identity that Databricks assumes to reach storage, keeping keys out of notebooks and cluster configurations.

Unity Catalog function attached to a table that decides, per user or group, which rows a query may return; the restriction applies to everyone who reads the table.

Unity Catalog function that substitutes a column's value at query time, for example hashed or partially hidden, for users who lack the privilege to see the real data.

View whose logic inspects the caller's identity or group membership and returns different rows or columns accordingly; it must live in a Unity Catalog schema to be secure.

Named store of secrets, backed by an encrypted vault or by Databricks itself, that notebooks and jobs read with dbutils.secrets so credentials never appear in code.

Record of workspace activity such as logins, permission changes, job runs, and table access, delivered to a system table or customer storage for security review and compliance.

YAML-defined project that bundles jobs, pipelines, and notebooks with their settings so they can be validated, deployed, and version-controlled consistently across environments.

Small key-value result that a workflow task sets for other tasks to read, used to pass parameters or drive conditional branching inside a job.

Dimension design that stores each historical version of a record as its own row with validity start and end dates and a current flag, preserving history across changes.

Pipeline dataset whose stored results are computed from an upstream query and refreshed on a schedule or trigger, so readers avoid recomputing expensive transformations.