Databricks Data Engineer Professional flashcards

32 cards. Read the term and the definition here. The deck at the top of this page is optional.

Pandas UDF

A vectorized Python UDF that exchanges data with the JVM through Apache Arrow in batches, avoiding per-row pickle serialization and typically running several times faster than a standard Python UDF.

MERGE INTO

A Delta SQL command that applies conditional inserts, updates, and deletes from a source to a target in one atomic transaction, and requires that each target row match at most one source row.

availableNow trigger

A Structured Streaming trigger that processes all currently available data as bounded micro-batches and then stops the query, which supports idempotent scheduled incremental loads.

Higher-order functions

SQL functions such as transform, filter, exists, and aggregate that accept a lambda and operate over ARRAY or MAP values without exploding rows into additional records.

Auto Loader

A Structured Streaming source using format cloudFiles that incrementally discovers new files in cloud storage, infers and evolves schema, and tracks processed files through a checkpoint.

COPY INTO

An idempotent SQL command that loads files from cloud storage into a Delta table while tracking which files were already processed, so reruns skip previously loaded files.

Rescued data column

The rescuedDataColumn option stores columns or records that fail schema parsing as JSON in an extra column instead of failing the load, preserving raw values for later repair.

Schema evolution modes

cloudFiles.schemaEvolutionMode settings: addNewColumns is the default and adds fields, failOnNewColumns raises an error, rescue stores mismatches, and none disables evolution.

Expectations

Data quality constraints in Lakeflow pipelines declared as EXPECT (condition) ON VIOLATION DROP ROW or FAIL UPDATE, with pass and fail counts recorded in the pipeline event log.

dropDuplicates

A DataFrame method that removes rows sharing identical values in the given subset of columns and keeps an arbitrary survivor, making the result non-deterministic without an ordering.

CHECK constraint

A Delta table constraint added with ALTER TABLE ADD CONSTRAINT that must evaluate true for every written row; a violation aborts the entire write transaction.

Schema enforcement

Delta rejects writes whose columns do not match the table, filling absent columns with null and refusing extra columns unless mergeSchema is explicitly enabled.

Delta Sharing

An open protocol for sharing Delta tables across organizations using short-lived bearer tokens or Databricks-to-Databricks sharing identifiers, without copying the underlying data.

Foreign catalog

A Unity Catalog object created with CREATE FOREIGN CATALOG over a Lakehouse Federation connection that exposes an external database read-only through catalog.schema.table naming.

Recipient

A Delta Sharing principal created with CREATE RECIPIENT that receives shares; open-sharing recipients get an activation link and credential file, while Databricks recipients get a sharing identifier.

DESCRIBE HISTORY

A command returning Delta transaction log entries for a table, including operation type, user, timestamp, and operationMetrics such as numOutputRows and numFiles.

System tables

Account-level read-only tables such as system.access.audit, system.billing.usage, and system.query.history that centralize audit, cost, and query observability data.

Lakehouse Monitoring

A Databricks feature that profiles tables or inference logs, writes profile and drift metric tables, and generates a dashboard for tracking data quality over time.

Liquid clustering

A Delta layout that clusters rows on declared columns and maintains the clustering incrementally, so clustering keys can be changed later without rewriting the table.

Z-ordering

OPTIMIZE ... ZORDER BY co-locates related values across files to improve multi-column data skipping; it cannot be combined with liquid clustering and must be rerun as data grows.

Adaptive Query Execution

A Spark runtime optimizer that re-plans a query between stages, coalescing small shuffle partitions, converting to broadcast joins, and splitting skewed partitions automatically.

VACUUM

A Delta operation that deletes data files no longer referenced by the transaction log once they exceed the retention interval, which defaults to seven days.

Row filter

A Unity Catalog function returning BOOLEAN attached with ALTER TABLE SET ROW FILTER, evaluated at query time to restrict which rows a given user may see.

Column mask

A Unity Catalog function attached with ALTER COLUMN SET MASK that transforms a column value at query time, for example returning a hash or null for unauthorized users.

Storage credential

A Unity Catalog object wrapping a cloud IAM role so external locations can be defined; users additionally need READ FILES or WRITE FILES on those locations.

information_schema

A read-only schema in each Unity Catalog catalog that exposes tables, columns, routines, and privilege grants as queryable metadata views for governance reporting.

Table lineage

Automatically captured provenance linking source and target tables, and their columns, for queries and pipelines; it is queryable in system.access.table_lineage.

Governed tag

An account-level tag with a controlled value that can be applied to catalogs, schemas, tables, or columns and referenced by attribute-based access policies.

Repair run

A job run option that re-executes only the failed or skipped tasks of a previous run while reusing the persisted outputs of tasks that already succeeded.

Declarative Automation Bundles

A project definition in databricks.yml that declares jobs, pipelines, and notebooks as resources and is validated and deployed per target with the Databricks CLI.

SCD Type 2

A dimension pattern that preserves history by inserting a new row with start and end timestamps plus a current flag each time a tracked attribute changes.

Medallion architecture

A layered lakehouse design where bronze holds raw ingested data, silver holds cleansed and conformed tables, and gold holds business-level aggregates for consumption.