Azure Databricks Data Engineer Associate flashcards

32 cards. Read the term and the definition here. The deck at the top of this page is optional.

Unity Catalog hierarchy

The object model is metastore, then catalog, then schema, then tables, views, volumes, and functions. Privileges granted at a higher level are inherited by all objects beneath it.

Job cluster versus all-purpose

A job cluster is created for a single job run and terminated when the run finishes, reducing cost. An all-purpose cluster stays running for interactive notebook and SQL work shared by users.

Photon engine

A vectorized C++ query engine that accelerates SQL and DataFrame operations on Databricks clusters. It is enabled per cluster and is most beneficial for large scans, joins, and aggregations.

Access modes on compute

Assigned mode restricts a cluster to one user or group. Shared mode supports multiple users with Unity Catalog and enforces language and feature restrictions for isolation.

Auto Loader schema evolution

Controlled by cloudFiles.schemaEvolutionMode with values addNewColumns, failOnNewColumns, rescue, and none. The schema location stores inferred schema and evolution history across runs.

COPY INTO idempotency

COPY INTO tracks which source files were already loaded, so rerunning the same statement does not duplicate rows. It suits periodic batch loads rather than continuous streaming ingestion.

Rescued data column

When rescuedDataColumn is enabled, fields that do not match the declared schema are stored in a _rescued_data column instead of failing the read, preserving malformed records.

JDBC parallel read options

partitionColumn, lowerBound, upperBound, and numPartitions split a JDBC source into parallel partitions. The partition column should be numeric, date, or timestamp and evenly distributed.

MERGE INTO clause order

Multiple WHEN MATCHED clauses are evaluated in the order written, and the first matching clause executes. WHEN NOT MATCHED BY SOURCE handles rows present only in the target table.

Delta transaction log

The _delta_log directory records every commit as JSON plus checkpoint Parquet files. It provides ACID guarantees, time travel, and the file list used by readers.

APPLY CHANGES for CDC

A pipeline operation that merges a change feed into a target table. SEQUENCE BY orders events, KEYS identifies rows, and deletes are applied with APPLY AS DELETE WHEN.

Liquid clustering

A Delta table layout where CLUSTER BY defines clustering keys instead of partitions. Keys can be changed without rewriting existing data, and OPTIMIZE maintains clustering automatically.

Repair run

A job run option that reruns only failed or skipped tasks plus their downstream dependencies. Completed tasks are not rerun, saving time and compute compared with a full run.

Dynamic value references

Templated values such as {{job.id}}, {{task.name}}, and {{job.start_time.iso_date}} resolved at runtime. They let task parameters use run-specific context without hardcoding values.

Job trigger types

Jobs can be triggered on a cron schedule, continuously, by file arrival, by a table update, or manually. Each trigger allows only one active run at a time.

Task dependency DAG

Tasks declare depends_on relationships forming a directed acyclic graph. Independent tasks run in parallel, and downstream tasks start only after all upstream tasks succeed.

Declarative Automation Bundles

YAML-defined projects in databricks.yml that describe jobs, pipelines, and notebooks. Commands include bundle validate, bundle deploy, and bundle run, with targets selecting environments.

Git folders in workspace

A workspace folder linked to a remote Git repository. Users commit, push, pull, and switch branches directly from the workspace, keeping notebooks version controlled.

Service principal deployment

A non-human identity used by CI/CD pipelines to authenticate to Databricks. It avoids dependence on personal tokens and supports OAuth machine-to-machine credentials.

Bundle targets

Named environments in databricks.yml, such as dev and prod, each with its own workspace host and variable overrides. Select one at deploy time with bundle deploy -t.

Spark UI Stages tab

Shows per-stage task counts, shuffle read and write bytes, spill to disk, and task duration distribution. Large spill or a few slow tasks indicate memory pressure or data skew.

Adaptive Query Execution

A Spark optimization that adjusts shuffle partitions, join strategies, and skew handling at runtime based on statistics. It is enabled by default in recent Databricks Runtime versions.

Skew join handling

Splits oversized partitions for skewed join keys when spark.sql.adaptive.skewJoin.enabled is true. Salting keys or broadcasting the small table are manual alternatives.

OPTIMIZE and Z-ORDER

OPTIMIZE compacts small files into larger ones. Z-ORDER co-locates related data on the specified columns so queries can skip more files during data skipping.

VACUUM retention

VACUUM deletes data files no longer referenced by the Delta log and older than the retention threshold, default seven days. Shortening it breaks time travel and concurrent readers.

System tables

Account-level Delta tables in the system catalog, including billing usage, compute clusters, and audit logs. They support cost analysis, monitoring, and compliance queries in SQL.

USE CATALOG and USE SCHEMA

Privileges required before a user can reference objects inside a catalog or schema. They are granted separately from SELECT and are needed even when table access is granted.

Row filter and column mask

Unity Catalog functions attached to a table that restrict visible rows or mask column values at query time. They apply per user or group without changing stored data.

Storage credential

A Unity Catalog object wrapping a cloud identity, such as an IAM role or managed identity, that authorizes access to storage. External locations pair a credential with a storage path.

Secret scopes

Containers for secrets backed by Databricks or a cloud key vault. Values are read with dbutils.secrets.get and are redacted when printed in notebook output.

SCIM provisioning

Synchronizes users and groups from an identity provider into the Databricks account. It centralizes identity management and supports automatic deprovisioning when accounts are removed.

Audit logs

Records of workspace and account activity such as logins, job runs, and permission changes. They are queried through system.access.audit or delivered to cloud storage for retention.