Azure Databricks Data Engineer Associate cheat sheet

Workspace objects: notebooks, folders, repos, jobs, clusters, SQL warehouses, and Unity Catalog securables all live in the workspace; notebooks support Python, SQL, Scala, and R with %md, %sql, %run, %pip, and %sh magics.

Cluster modes: all-purpose clusters are interactive and shared; job clusters are created per run and terminated at completion; single-node clusters have no workers; high-concurrency clusters support table ACLs and many concurrent users.

Runtime version format is DBR ..x-scala; Databricks Runtime ML adds MLflow, GPU libraries, and AutoML; Photon is a C++ vectorized query engine enabled per cluster, best for SQL and DataFrame workloads.

Access modes: assigned (formerly single user) for one user or group; shared (formerly shared) for Unity Catalog with multiple users and supported languages; the access mode restricts which features and languages are available.

Delta Lake is the default table format; managed tables store data in the metastore-managed location, external tables reference a user-specified path; Unity Catalog is the default metastore for new workspaces.

Compute sizing: driver and worker node types are independent; autoscaling sets min and max workers; spot instances reduce cost but can be reclaimed; cluster policies constrain node types, tags, and autotermination.

Notebook results are attached to the cluster; %run shares variables and functions from another notebook; dbutils.notebook.run executes a notebook as a separate job with parameters and returns a value.

Auto Loader (cloudFiles) incrementally ingests new files from cloud storage using directory listing or file notification mode; schema inference and schema evolution are controlled by cloudFiles.schemaLocation and cloudFiles.schemaEvolutionMode.

COPY INTO is idempotent for a given source file set and is best for periodic batch loads of thousands of files; Auto Loader scales better for continuous or very large file streams.

read_files (formerly the format-agnostic reader) reads CSV, JSON, Parquet, Avro, ORC, and text directly from a path or volume and supports schema hints and rescued data.

Ingestion patterns: batch (COPY INTO, JDBC, spark.read), streaming (Auto Loader, Kafka, Kinesis, Delta Live Tables streaming tables), and federated queries against external databases via Lakehouse Federation.

Schema evolution modes include addNewColumns, failOnNewColumns, rescue, and none; rescued data is captured in a _rescued_data column when rescuedDataColumn is enabled.

JDBC ingestion uses spark.read.format("jdbc") with url, dbtable, user, and password; partitioning options partitionColumn, lowerBound, upperBound, and numPartitions control parallel reads.

File formats: Parquet is columnar and splittable; Avro is row-based with schema evolution; CSV and JSON need explicit schemas for performance; Delta is the recommended target for all ingestion.

Ingestion into Unity Catalog volumes uses /Volumes//// paths; volumes can be managed or external and are governed by catalog privileges.

Delta Lake ACID transactions use a _delta_log transaction log; OPTIMIZE compacts small files, Z-ORDER co-locates data on columns, and VACUUM removes files no longer referenced by the log.

MERGE INTO supports WHEN MATCHED, WHEN NOT MATCHED, WHEN NOT MATCHED BY SOURCE, and WHEN MATCHED THEN DELETE clauses; multiple matched clauses are evaluated in order.

Medallion architecture: bronze holds raw ingested data, silver holds cleaned and conformed data, gold holds business-level aggregates; each layer is a Delta table governed in Unity Catalog.

Delta Live Tables (Lakeflow Declarative Pipelines) define streaming tables with CREATE OR REFRESH STREAMING TABLE and materialized views with CREATE OR REFRESH MATERIALIZED VIEW, plus APPLY CHANGES for CDC.

Window functions such as ROW_NUMBER, RANK, LAG, LEAD, and SUM OVER require an OVER clause with PARTITION BY and ORDER BY; use them to deduplicate and compute running totals.

Data quality constraints use EXPECT ... ON VIOLATION DROP ROW or FAIL UPDATE; expectations are declared in pipeline code and reported in pipeline event logs.

Liquid clustering replaces partitioning and Z-ORDER for many tables; CLUSTER BY defines clustering keys that can be changed without rewriting data.

Change Data Capture with APPLY CHANGES INTO handles inserts, updates, and deletes from a change feed using SEQUENCE BY and KEYS clauses.

A Lakeflow job is a collection of tasks with dependencies; task types include notebook, Python script, SQL, dbt, pipeline, and run-job tasks, and tasks can run in parallel when they have no dependency.

Trigger types: scheduled (cron with timezone), continuous, file arrival, table update, and manual; a job can have multiple triggers but only one active run per trigger at a time.

Job parameters are passed to tasks via job parameters and task parameters; dynamic value references such as {{job.id}}, {{task.name}}, and {{job.start_time.iso_date}} are resolved at runtime.

Repair run reruns only failed or skipped tasks and their downstream dependencies; a full run reruns all tasks; both preserve the original run history.

Task dependencies form a DAG; each task runs on its own compute (job cluster, existing cluster, SQL warehouse, or serverless) and can have retries, timeouts, and notifications.

Notifications can be configured per job or per task for on-start, on-success, and on-failure events, delivered via email, webhook, or the Jobs API.

Job compute: serverless compute is available for notebooks and SQL tasks; job clusters terminate after the run; cluster reuse within a job is configured with the same cluster definition.

Databricks Repos (Git folders) link a workspace folder to a Git provider; supported operations include commit, push, pull, branch checkout, and create pull request.

Declarative Automation Bundles (formerly Databricks Asset Bundles) define jobs, pipelines, notebooks, and cluster configs in databricks.yml plus resource YAML files; validate with databricks bundle validate and deploy with databricks bundle deploy.

Bundle targets map to environments (dev, staging, prod) with per-target workspace hosts and variables; use databricks bundle deploy -t to select a target.

The Databricks CLI (v0.2xx and later) authenticates with a personal access token, OAuth, or a service principal; databricks auth login configures a profile.

CI/CD pipeline stages: build and unit test, validate bundle, deploy to dev, run integration tests, promote to prod; use service principals for automated deployments rather than personal tokens.

Terraform provider databricks is an alternative to bundles for provisioning workspaces, clusters, jobs, and Unity Catalog objects; state should be stored remotely.

Secrets for CI/CD should be stored in the CI system (GitHub Actions secrets, Azure DevOps variable groups) and referenced as environment variables, never committed to the repo.

Spark UI tabs: Jobs, Stages, Tasks, SQL, Storage, and Executors; the Stages tab shows shuffle read/write, spill, and task skew, which indicate skew or insufficient memory.

Common failure causes: OOM on the driver from collect() or large broadcast, OOM on executors from skewed partitions, and shuffle failures from disk space or lost executors.

Skew mitigation: salting keys, AQE skew join handling (spark.sql.adaptive.skewJoin.enabled), broadcast joins for small tables (spark.sql.autoBroadcastJoinThreshold), and repartitioning.

Adaptive Query Execution (AQE) coalesces shuffle partitions, converts sort-merge joins to broadcast joins, and handles skew; enabled by default in recent DBR versions.

Diagnostic logs: cluster log delivery to a cloud storage path, driver logs, and the Spark event log; pipeline event logs capture data quality metrics and lineage.

Query profiling: the Query Profile in the SQL editor shows operator-level time, rows, and spill; look for long scans, exploding joins, and excessive shuffles.

Delta optimization: OPTIMIZE with Z-ORDER or liquid clustering for file layout, VACUUM to reclaim storage, and ANALYZE TABLE to update statistics for the optimizer.

Monitoring: system tables (system.billing.usage, system.compute.clusters, system.access.audit) in Unity Catalog provide account-level observability; job run history and alerts are in the Jobs UI.

Unity Catalog hierarchy: metastore > catalog > schema > table/view/volume/function; privileges are granted at any level and inherited downward; USE CATALOG and USE SCHEMA are required to access objects.

Privilege types include SELECT, MODIFY, CREATE, USAGE, ALL PRIVILEGES, and ownership; owners and metastore admins can grant privileges; GRANT and REVOKE are the SQL commands.

Data isolation: dynamic views and row filter and column mask functions restrict access at query time; these are defined on tables and applied per user or group.

External locations require a storage credential backed by an IAM role or managed identity; external tables and external volumes must be under an external location.

Authentication: personal access tokens, OAuth machine-to-machine with service principals, and SSO via the identity provider; SCIM provisioning syncs users and groups from the IdP.

Audit logs record workspace and account activity; system.access.audit in Unity Catalog exposes audit events for querying; cluster log delivery and diagnostic logs support compliance.

Secrets are stored in secret scopes (Databricks-backed or Azure Key Vault-backed) and accessed with dbutils.secrets.get; secret values are redacted in notebook output.

Table ACLs and Unity Catalog grants differ: Unity Catalog is the recommended model and provides centralized governance across workspaces in an account.

Workspace objects: notebooks, folders, repos, jobs, clusters, SQL warehouses, and Unity Catalog securables all live in the workspace; notebooks support Python, SQL, Scala, and R with %md, %sql, %run, %pip, and %sh magics.

Cluster modes: all-purpose clusters are interactive and shared; job clusters are created per run and terminated at completion; single-node clusters have no workers; high-concurrency clusters support table ACLs and many concurrent users.

Runtime version format is DBR ..x-scala; Databricks Runtime ML adds MLflow, GPU libraries, and AutoML; Photon is a C++ vectorized query engine enabled per cluster, best for SQL and DataFrame workloads.

Access modes: assigned (formerly single user) for one user or group; shared (formerly shared) for Unity Catalog with multiple users and supported languages; the access mode restricts which features and languages are available.

Delta Lake is the default table format; managed tables store data in the metastore-managed location, external tables reference a user-specified path; Unity Catalog is the default metastore for new workspaces.

Compute sizing: driver and worker node types are independent; autoscaling sets min and max workers; spot instances reduce cost but can be reclaimed; cluster policies constrain node types, tags, and autotermination.

Notebook results are attached to the cluster; %run shares variables and functions from another notebook; dbutils.notebook.run executes a notebook as a separate job with parameters and returns a value.

Auto Loader (cloudFiles) incrementally ingests new files from cloud storage using directory listing or file notification mode; schema inference and schema evolution are controlled by cloudFiles.schemaLocation and cloudFiles.schemaEvolutionMode.

COPY INTO is idempotent for a given source file set and is best for periodic batch loads of thousands of files; Auto Loader scales better for continuous or very large file streams.

read_files (formerly the format-agnostic reader) reads CSV, JSON, Parquet, Avro, ORC, and text directly from a path or volume and supports schema hints and rescued data.

Ingestion patterns: batch (COPY INTO, JDBC, spark.read), streaming (Auto Loader, Kafka, Kinesis, Delta Live Tables streaming tables), and federated queries against external databases via Lakehouse Federation.

Schema evolution modes include addNewColumns, failOnNewColumns, rescue, and none; rescued data is captured in a _rescued_data column when rescuedDataColumn is enabled.

JDBC ingestion uses spark.read.format("jdbc") with url, dbtable, user, and password; partitioning options partitionColumn, lowerBound, upperBound, and numPartitions control parallel reads.

File formats: Parquet is columnar and splittable; Avro is row-based with schema evolution; CSV and JSON need explicit schemas for performance; Delta is the recommended target for all ingestion.

Ingestion into Unity Catalog volumes uses /Volumes//// paths; volumes can be managed or external and are governed by catalog privileges.

Delta Lake ACID transactions use a _delta_log transaction log; OPTIMIZE compacts small files, Z-ORDER co-locates data on columns, and VACUUM removes files no longer referenced by the log.

MERGE INTO supports WHEN MATCHED, WHEN NOT MATCHED, WHEN NOT MATCHED BY SOURCE, and WHEN MATCHED THEN DELETE clauses; multiple matched clauses are evaluated in order.

Medallion architecture: bronze holds raw ingested data, silver holds cleaned and conformed data, gold holds business-level aggregates; each layer is a Delta table governed in Unity Catalog.

Delta Live Tables (Lakeflow Declarative Pipelines) define streaming tables with CREATE OR REFRESH STREAMING TABLE and materialized views with CREATE OR REFRESH MATERIALIZED VIEW, plus APPLY CHANGES for CDC.

Window functions such as ROW_NUMBER, RANK, LAG, LEAD, and SUM OVER require an OVER clause with PARTITION BY and ORDER BY; use them to deduplicate and compute running totals.

Data quality constraints use EXPECT ... ON VIOLATION DROP ROW or FAIL UPDATE; expectations are declared in pipeline code and reported in pipeline event logs.

Liquid clustering replaces partitioning and Z-ORDER for many tables; CLUSTER BY defines clustering keys that can be changed without rewriting data.

Change Data Capture with APPLY CHANGES INTO handles inserts, updates, and deletes from a change feed using SEQUENCE BY and KEYS clauses.

A Lakeflow job is a collection of tasks with dependencies; task types include notebook, Python script, SQL, dbt, pipeline, and run-job tasks, and tasks can run in parallel when they have no dependency.

Trigger types: scheduled (cron with timezone), continuous, file arrival, table update, and manual; a job can have multiple triggers but only one active run per trigger at a time.

Job parameters are passed to tasks via job parameters and task parameters; dynamic value references such as {{job.id}}, {{task.name}}, and {{job.start_time.iso_date}} are resolved at runtime.

Repair run reruns only failed or skipped tasks and their downstream dependencies; a full run reruns all tasks; both preserve the original run history.

Task dependencies form a DAG; each task runs on its own compute (job cluster, existing cluster, SQL warehouse, or serverless) and can have retries, timeouts, and notifications.

Notifications can be configured per job or per task for on-start, on-success, and on-failure events, delivered via email, webhook, or the Jobs API.

Job compute: serverless compute is available for notebooks and SQL tasks; job clusters terminate after the run; cluster reuse within a job is configured with the same cluster definition.

Databricks Repos (Git folders) link a workspace folder to a Git provider; supported operations include commit, push, pull, branch checkout, and create pull request.

Declarative Automation Bundles (formerly Databricks Asset Bundles) define jobs, pipelines, notebooks, and cluster configs in databricks.yml plus resource YAML files; validate with databricks bundle validate and deploy with databricks bundle deploy.

Bundle targets map to environments (dev, staging, prod) with per-target workspace hosts and variables; use databricks bundle deploy -t to select a target.

The Databricks CLI (v0.2xx and later) authenticates with a personal access token, OAuth, or a service principal; databricks auth login configures a profile.

CI/CD pipeline stages: build and unit test, validate bundle, deploy to dev, run integration tests, promote to prod; use service principals for automated deployments rather than personal tokens.

Terraform provider databricks is an alternative to bundles for provisioning workspaces, clusters, jobs, and Unity Catalog objects; state should be stored remotely.

Secrets for CI/CD should be stored in the CI system (GitHub Actions secrets, Azure DevOps variable groups) and referenced as environment variables, never committed to the repo.

Spark UI tabs: Jobs, Stages, Tasks, SQL, Storage, and Executors; the Stages tab shows shuffle read/write, spill, and task skew, which indicate skew or insufficient memory.

Common failure causes: OOM on the driver from collect() or large broadcast, OOM on executors from skewed partitions, and shuffle failures from disk space or lost executors.

Skew mitigation: salting keys, AQE skew join handling (spark.sql.adaptive.skewJoin.enabled), broadcast joins for small tables (spark.sql.autoBroadcastJoinThreshold), and repartitioning.

Adaptive Query Execution (AQE) coalesces shuffle partitions, converts sort-merge joins to broadcast joins, and handles skew; enabled by default in recent DBR versions.

Diagnostic logs: cluster log delivery to a cloud storage path, driver logs, and the Spark event log; pipeline event logs capture data quality metrics and lineage.

Query profiling: the Query Profile in the SQL editor shows operator-level time, rows, and spill; look for long scans, exploding joins, and excessive shuffles.

Delta optimization: OPTIMIZE with Z-ORDER or liquid clustering for file layout, VACUUM to reclaim storage, and ANALYZE TABLE to update statistics for the optimizer.

Monitoring: system tables (system.billing.usage, system.compute.clusters, system.access.audit) in Unity Catalog provide account-level observability; job run history and alerts are in the Jobs UI.

Unity Catalog hierarchy: metastore > catalog > schema > table/view/volume/function; privileges are granted at any level and inherited downward; USE CATALOG and USE SCHEMA are required to access objects.

Privilege types include SELECT, MODIFY, CREATE, USAGE, ALL PRIVILEGES, and ownership; owners and metastore admins can grant privileges; GRANT and REVOKE are the SQL commands.

Data isolation: dynamic views and row filter and column mask functions restrict access at query time; these are defined on tables and applied per user or group.

External locations require a storage credential backed by an IAM role or managed identity; external tables and external volumes must be under an external location.

Authentication: personal access tokens, OAuth machine-to-machine with service principals, and SSO via the identity provider; SCIM provisioning syncs users and groups from the IdP.

Audit logs record workspace and account activity; system.access.audit in Unity Catalog exposes audit events for querying; cluster log delivery and diagnostic logs support compliance.

Secrets are stored in secret scopes (Databricks-backed or Azure Key Vault-backed) and accessed with dbutils.secrets.get; secret values are redacted in notebook output.

Table ACLs and Unity Catalog grants differ: Unity Catalog is the recommended model and provides centralized governance across workspaces in an account.

Workspace objects: notebooks, folders, repos, jobs, clusters, SQL warehouses, and Unity Catalog securables all live in the workspace; notebooks support Python, SQL, Scala, and R with %md, %sql, %run, %pip, and %sh magics.

Cluster modes: all-purpose clusters are interactive and shared; job clusters are created per run and terminated at completion; single-node clusters have no workers; high-concurrency clusters support table ACLs and many concurrent users.

Runtime version format is DBR ..x-scala; Databricks Runtime ML adds MLflow, GPU libraries, and AutoML; Photon is a C++ vectorized query engine enabled per cluster, best for SQL and DataFrame workloads.

Access modes: assigned (formerly single user) for one user or group; shared (formerly shared) for Unity Catalog with multiple users and supported languages; the access mode restricts which features and languages are available.

Delta Lake is the default table format; managed tables store data in the metastore-managed location, external tables reference a user-specified path; Unity Catalog is the default metastore for new workspaces.

Compute sizing: driver and worker node types are independent; autoscaling sets min and max workers; spot instances reduce cost but can be reclaimed; cluster policies constrain node types, tags, and autotermination.

Notebook results are attached to the cluster; %run shares variables and functions from another notebook; dbutils.notebook.run executes a notebook as a separate job with parameters and returns a value.

Auto Loader (cloudFiles) incrementally ingests new files from cloud storage using directory listing or file notification mode; schema inference and schema evolution are controlled by cloudFiles.schemaLocation and cloudFiles.schemaEvolutionMode.

COPY INTO is idempotent for a given source file set and is best for periodic batch loads of thousands of files; Auto Loader scales better for continuous or very large file streams.

read_files (formerly the format-agnostic reader) reads CSV, JSON, Parquet, Avro, ORC, and text directly from a path or volume and supports schema hints and rescued data.

Ingestion patterns: batch (COPY INTO, JDBC, spark.read), streaming (Auto Loader, Kafka, Kinesis, Delta Live Tables streaming tables), and federated queries against external databases via Lakehouse Federation.

Schema evolution modes include addNewColumns, failOnNewColumns, rescue, and none; rescued data is captured in a _rescued_data column when rescuedDataColumn is enabled.

JDBC ingestion uses spark.read.format("jdbc") with url, dbtable, user, and password; partitioning options partitionColumn, lowerBound, upperBound, and numPartitions control parallel reads.

File formats: Parquet is columnar and splittable; Avro is row-based with schema evolution; CSV and JSON need explicit schemas for performance; Delta is the recommended target for all ingestion.

Ingestion into Unity Catalog volumes uses /Volumes//// paths; volumes can be managed or external and are governed by catalog privileges.

Delta Lake ACID transactions use a _delta_log transaction log; OPTIMIZE compacts small files, Z-ORDER co-locates data on columns, and VACUUM removes files no longer referenced by the log.

MERGE INTO supports WHEN MATCHED, WHEN NOT MATCHED, WHEN NOT MATCHED BY SOURCE, and WHEN MATCHED THEN DELETE clauses; multiple matched clauses are evaluated in order.

Medallion architecture: bronze holds raw ingested data, silver holds cleaned and conformed data, gold holds business-level aggregates; each layer is a Delta table governed in Unity Catalog.

Delta Live Tables (Lakeflow Declarative Pipelines) define streaming tables with CREATE OR REFRESH STREAMING TABLE and materialized views with CREATE OR REFRESH MATERIALIZED VIEW, plus APPLY CHANGES for CDC.

Window functions such as ROW_NUMBER, RANK, LAG, LEAD, and SUM OVER require an OVER clause with PARTITION BY and ORDER BY; use them to deduplicate and compute running totals.

Data quality constraints use EXPECT ... ON VIOLATION DROP ROW or FAIL UPDATE; expectations are declared in pipeline code and reported in pipeline event logs.

Liquid clustering replaces partitioning and Z-ORDER for many tables; CLUSTER BY defines clustering keys that can be changed without rewriting data.

Change Data Capture with APPLY CHANGES INTO handles inserts, updates, and deletes from a change feed using SEQUENCE BY and KEYS clauses.

A Lakeflow job is a collection of tasks with dependencies; task types include notebook, Python script, SQL, dbt, pipeline, and run-job tasks, and tasks can run in parallel when they have no dependency.

Trigger types: scheduled (cron with timezone), continuous, file arrival, table update, and manual; a job can have multiple triggers but only one active run per trigger at a time.

Job parameters are passed to tasks via job parameters and task parameters; dynamic value references such as {{job.id}}, {{task.name}}, and {{job.start_time.iso_date}} are resolved at runtime.

Repair run reruns only failed or skipped tasks and their downstream dependencies; a full run reruns all tasks; both preserve the original run history.

Task dependencies form a DAG; each task runs on its own compute (job cluster, existing cluster, SQL warehouse, or serverless) and can have retries, timeouts, and notifications.

Notifications can be configured per job or per task for on-start, on-success, and on-failure events, delivered via email, webhook, or the Jobs API.

Job compute: serverless compute is available for notebooks and SQL tasks; job clusters terminate after the run; cluster reuse within a job is configured with the same cluster definition.

Databricks Repos (Git folders) link a workspace folder to a Git provider; supported operations include commit, push, pull, branch checkout, and create pull request.

Declarative Automation Bundles (formerly Databricks Asset Bundles) define jobs, pipelines, notebooks, and cluster configs in databricks.yml plus resource YAML files; validate with databricks bundle validate and deploy with databricks bundle deploy.

Bundle targets map to environments (dev, staging, prod) with per-target workspace hosts and variables; use databricks bundle deploy -t to select a target.

The Databricks CLI (v0.2xx and later) authenticates with a personal access token, OAuth, or a service principal; databricks auth login configures a profile.

CI/CD pipeline stages: build and unit test, validate bundle, deploy to dev, run integration tests, promote to prod; use service principals for automated deployments rather than personal tokens.

Terraform provider databricks is an alternative to bundles for provisioning workspaces, clusters, jobs, and Unity Catalog objects; state should be stored remotely.

Secrets for CI/CD should be stored in the CI system (GitHub Actions secrets, Azure DevOps variable groups) and referenced as environment variables, never committed to the repo.

Spark UI tabs: Jobs, Stages, Tasks, SQL, Storage, and Executors; the Stages tab shows shuffle read/write, spill, and task skew, which indicate skew or insufficient memory.

Common failure causes: OOM on the driver from collect() or large broadcast, OOM on executors from skewed partitions, and shuffle failures from disk space or lost executors.

Skew mitigation: salting keys, AQE skew join handling (spark.sql.adaptive.skewJoin.enabled), broadcast joins for small tables (spark.sql.autoBroadcastJoinThreshold), and repartitioning.

Adaptive Query Execution (AQE) coalesces shuffle partitions, converts sort-merge joins to broadcast joins, and handles skew; enabled by default in recent DBR versions.

Diagnostic logs: cluster log delivery to a cloud storage path, driver logs, and the Spark event log; pipeline event logs capture data quality metrics and lineage.

Query profiling: the Query Profile in the SQL editor shows operator-level time, rows, and spill; look for long scans, exploding joins, and excessive shuffles.

Delta optimization: OPTIMIZE with Z-ORDER or liquid clustering for file layout, VACUUM to reclaim storage, and ANALYZE TABLE to update statistics for the optimizer.

Monitoring: system tables (system.billing.usage, system.compute.clusters, system.access.audit) in Unity Catalog provide account-level observability; job run history and alerts are in the Jobs UI.

Unity Catalog hierarchy: metastore > catalog > schema > table/view/volume/function; privileges are granted at any level and inherited downward; USE CATALOG and USE SCHEMA are required to access objects.

Privilege types include SELECT, MODIFY, CREATE, USAGE, ALL PRIVILEGES, and ownership; owners and metastore admins can grant privileges; GRANT and REVOKE are the SQL commands.

Data isolation: dynamic views and row filter and column mask functions restrict access at query time; these are defined on tables and applied per user or group.

External locations require a storage credential backed by an IAM role or managed identity; external tables and external volumes must be under an external location.

Authentication: personal access tokens, OAuth machine-to-machine with service principals, and SSO via the identity provider; SCIM provisioning syncs users and groups from the IdP.

Audit logs record workspace and account activity; system.access.audit in Unity Catalog exposes audit events for querying; cluster log delivery and diagnostic logs support compliance.

Secrets are stored in secret scopes (Databricks-backed or Azure Key Vault-backed) and accessed with dbutils.secrets.get; secret values are redacted in notebook output.

Table ACLs and Unity Catalog grants differ: Unity Catalog is the recommended model and provides centralized governance across workspaces in an account.

Workspace objects: notebooks, folders, repos, jobs, clusters, SQL warehouses, and Unity Catalog securables all live in the workspace; notebooks support Python, SQL, Scala, and R with %md, %sql, %run, %pip, and %sh magics.

Cluster modes: all-purpose clusters are interactive and shared; job clusters are created per run and terminated at completion; single-node clusters have no workers; high-concurrency clusters support table ACLs and many concurrent users.

Runtime version format is DBR ..x-scala; Databricks Runtime ML adds MLflow, GPU libraries, and AutoML; Photon is a C++ vectorized query engine enabled per cluster, best for SQL and DataFrame workloads.

Access modes: assigned (formerly single user) for one user or group; shared (formerly shared) for Unity Catalog with multiple users and supported languages; the access mode restricts which features and languages are available.

Delta Lake is the default table format; managed tables store data in the metastore-managed location, external tables reference a user-specified path; Unity Catalog is the default metastore for new workspaces.

Compute sizing: driver and worker node types are independent; autoscaling sets min and max workers; spot instances reduce cost but can be reclaimed; cluster policies constrain node types, tags, and autotermination.

Notebook results are attached to the cluster; %run shares variables and functions from another notebook; dbutils.notebook.run executes a notebook as a separate job with parameters and returns a value.

Auto Loader (cloudFiles) incrementally ingests new files from cloud storage using directory listing or file notification mode; schema inference and schema evolution are controlled by cloudFiles.schemaLocation and cloudFiles.schemaEvolutionMode.

COPY INTO is idempotent for a given source file set and is best for periodic batch loads of thousands of files; Auto Loader scales better for continuous or very large file streams.

read_files (formerly the format-agnostic reader) reads CSV, JSON, Parquet, Avro, ORC, and text directly from a path or volume and supports schema hints and rescued data.

Ingestion patterns: batch (COPY INTO, JDBC, spark.read), streaming (Auto Loader, Kafka, Kinesis, Delta Live Tables streaming tables), and federated queries against external databases via Lakehouse Federation.

Schema evolution modes include addNewColumns, failOnNewColumns, rescue, and none; rescued data is captured in a _rescued_data column when rescuedDataColumn is enabled.

JDBC ingestion uses spark.read.format("jdbc") with url, dbtable, user, and password; partitioning options partitionColumn, lowerBound, upperBound, and numPartitions control parallel reads.

File formats: Parquet is columnar and splittable; Avro is row-based with schema evolution; CSV and JSON need explicit schemas for performance; Delta is the recommended target for all ingestion.

Ingestion into Unity Catalog volumes uses /Volumes//// paths; volumes can be managed or external and are governed by catalog privileges.

Delta Lake ACID transactions use a _delta_log transaction log; OPTIMIZE compacts small files, Z-ORDER co-locates data on columns, and VACUUM removes files no longer referenced by the log.

MERGE INTO supports WHEN MATCHED, WHEN NOT MATCHED, WHEN NOT MATCHED BY SOURCE, and WHEN MATCHED THEN DELETE clauses; multiple matched clauses are evaluated in order.

Medallion architecture: bronze holds raw ingested data, silver holds cleaned and conformed data, gold holds business-level aggregates; each layer is a Delta table governed in Unity Catalog.

Delta Live Tables (Lakeflow Declarative Pipelines) define streaming tables with CREATE OR REFRESH STREAMING TABLE and materialized views with CREATE OR REFRESH MATERIALIZED VIEW, plus APPLY CHANGES for CDC.

Window functions such as ROW_NUMBER, RANK, LAG, LEAD, and SUM OVER require an OVER clause with PARTITION BY and ORDER BY; use them to deduplicate and compute running totals.

Data quality constraints use EXPECT ... ON VIOLATION DROP ROW or FAIL UPDATE; expectations are declared in pipeline code and reported in pipeline event logs.

Liquid clustering replaces partitioning and Z-ORDER for many tables; CLUSTER BY defines clustering keys that can be changed without rewriting data.

Change Data Capture with APPLY CHANGES INTO handles inserts, updates, and deletes from a change feed using SEQUENCE BY and KEYS clauses.

A Lakeflow job is a collection of tasks with dependencies; task types include notebook, Python script, SQL, dbt, pipeline, and run-job tasks, and tasks can run in parallel when they have no dependency.

Trigger types: scheduled (cron with timezone), continuous, file arrival, table update, and manual; a job can have multiple triggers but only one active run per trigger at a time.

Job parameters are passed to tasks via job parameters and task parameters; dynamic value references such as {{job.id}}, {{task.name}}, and {{job.start_time.iso_date}} are resolved at runtime.

Repair run reruns only failed or skipped tasks and their downstream dependencies; a full run reruns all tasks; both preserve the original run history.

Task dependencies form a DAG; each task runs on its own compute (job cluster, existing cluster, SQL warehouse, or serverless) and can have retries, timeouts, and notifications.

Notifications can be configured per job or per task for on-start, on-success, and on-failure events, delivered via email, webhook, or the Jobs API.

Job compute: serverless compute is available for notebooks and SQL tasks; job clusters terminate after the run; cluster reuse within a job is configured with the same cluster definition.

Databricks Repos (Git folders) link a workspace folder to a Git provider; supported operations include commit, push, pull, branch checkout, and create pull request.

Declarative Automation Bundles (formerly Databricks Asset Bundles) define jobs, pipelines, notebooks, and cluster configs in databricks.yml plus resource YAML files; validate with databricks bundle validate and deploy with databricks bundle deploy.

Bundle targets map to environments (dev, staging, prod) with per-target workspace hosts and variables; use databricks bundle deploy -t to select a target.

The Databricks CLI (v0.2xx and later) authenticates with a personal access token, OAuth, or a service principal; databricks auth login configures a profile.

CI/CD pipeline stages: build and unit test, validate bundle, deploy to dev, run integration tests, promote to prod; use service principals for automated deployments rather than personal tokens.

Terraform provider databricks is an alternative to bundles for provisioning workspaces, clusters, jobs, and Unity Catalog objects; state should be stored remotely.

Secrets for CI/CD should be stored in the CI system (GitHub Actions secrets, Azure DevOps variable groups) and referenced as environment variables, never committed to the repo.

Spark UI tabs: Jobs, Stages, Tasks, SQL, Storage, and Executors; the Stages tab shows shuffle read/write, spill, and task skew, which indicate skew or insufficient memory.

Common failure causes: OOM on the driver from collect() or large broadcast, OOM on executors from skewed partitions, and shuffle failures from disk space or lost executors.

Skew mitigation: salting keys, AQE skew join handling (spark.sql.adaptive.skewJoin.enabled), broadcast joins for small tables (spark.sql.autoBroadcastJoinThreshold), and repartitioning.

Adaptive Query Execution (AQE) coalesces shuffle partitions, converts sort-merge joins to broadcast joins, and handles skew; enabled by default in recent DBR versions.

Diagnostic logs: cluster log delivery to a cloud storage path, driver logs, and the Spark event log; pipeline event logs capture data quality metrics and lineage.

Query profiling: the Query Profile in the SQL editor shows operator-level time, rows, and spill; look for long scans, exploding joins, and excessive shuffles.

Delta optimization: OPTIMIZE with Z-ORDER or liquid clustering for file layout, VACUUM to reclaim storage, and ANALYZE TABLE to update statistics for the optimizer.

Monitoring: system tables (system.billing.usage, system.compute.clusters, system.access.audit) in Unity Catalog provide account-level observability; job run history and alerts are in the Jobs UI.

Unity Catalog hierarchy: metastore > catalog > schema > table/view/volume/function; privileges are granted at any level and inherited downward; USE CATALOG and USE SCHEMA are required to access objects.

Privilege types include SELECT, MODIFY, CREATE, USAGE, ALL PRIVILEGES, and ownership; owners and metastore admins can grant privileges; GRANT and REVOKE are the SQL commands.

Data isolation: dynamic views and row filter and column mask functions restrict access at query time; these are defined on tables and applied per user or group.

External locations require a storage credential backed by an IAM role or managed identity; external tables and external volumes must be under an external location.

Authentication: personal access tokens, OAuth machine-to-machine with service principals, and SSO via the identity provider; SCIM provisioning syncs users and groups from the IdP.

Audit logs record workspace and account activity; system.access.audit in Unity Catalog exposes audit events for querying; cluster log delivery and diagnostic logs support compliance.

Secrets are stored in secret scopes (Databricks-backed or Azure Key Vault-backed) and accessed with dbutils.secrets.get; secret values are redacted in notebook output.

Table ACLs and Unity Catalog grants differ: Unity Catalog is the recommended model and provides centralized governance across workspaces in an account.

Workspace objects: notebooks, folders, repos, jobs, clusters, SQL warehouses, and Unity Catalog securables all live in the workspace; notebooks support Python, SQL, Scala, and R with %md, %sql, %run, %pip, and %sh magics.

Cluster modes: all-purpose clusters are interactive and shared; job clusters are created per run and terminated at completion; single-node clusters have no workers; high-concurrency clusters support table ACLs and many concurrent users.

Runtime version format is DBR ..x-scala; Databricks Runtime ML adds MLflow, GPU libraries, and AutoML; Photon is a C++ vectorized query engine enabled per cluster, best for SQL and DataFrame workloads.

Access modes: assigned (formerly single user) for one user or group; shared (formerly shared) for Unity Catalog with multiple users and supported languages; the access mode restricts which features and languages are available.

Delta Lake is the default table format; managed tables store data in the metastore-managed location, external tables reference a user-specified path; Unity Catalog is the default metastore for new workspaces.

Compute sizing: driver and worker node types are independent; autoscaling sets min and max workers; spot instances reduce cost but can be reclaimed; cluster policies constrain node types, tags, and autotermination.

Notebook results are attached to the cluster; %run shares variables and functions from another notebook; dbutils.notebook.run executes a notebook as a separate job with parameters and returns a value.

Auto Loader (cloudFiles) incrementally ingests new files from cloud storage using directory listing or file notification mode; schema inference and schema evolution are controlled by cloudFiles.schemaLocation and cloudFiles.schemaEvolutionMode.

COPY INTO is idempotent for a given source file set and is best for periodic batch loads of thousands of files; Auto Loader scales better for continuous or very large file streams.

read_files (formerly the format-agnostic reader) reads CSV, JSON, Parquet, Avro, ORC, and text directly from a path or volume and supports schema hints and rescued data.

Ingestion patterns: batch (COPY INTO, JDBC, spark.read), streaming (Auto Loader, Kafka, Kinesis, Delta Live Tables streaming tables), and federated queries against external databases via Lakehouse Federation.

Schema evolution modes include addNewColumns, failOnNewColumns, rescue, and none; rescued data is captured in a _rescued_data column when rescuedDataColumn is enabled.

JDBC ingestion uses spark.read.format("jdbc") with url, dbtable, user, and password; partitioning options partitionColumn, lowerBound, upperBound, and numPartitions control parallel reads.

File formats: Parquet is columnar and splittable; Avro is row-based with schema evolution; CSV and JSON need explicit schemas for performance; Delta is the recommended target for all ingestion.

Ingestion into Unity Catalog volumes uses /Volumes//// paths; volumes can be managed or external and are governed by catalog privileges.

Delta Lake ACID transactions use a _delta_log transaction log; OPTIMIZE compacts small files, Z-ORDER co-locates data on columns, and VACUUM removes files no longer referenced by the log.

MERGE INTO supports WHEN MATCHED, WHEN NOT MATCHED, WHEN NOT MATCHED BY SOURCE, and WHEN MATCHED THEN DELETE clauses; multiple matched clauses are evaluated in order.

Medallion architecture: bronze holds raw ingested data, silver holds cleaned and conformed data, gold holds business-level aggregates; each layer is a Delta table governed in Unity Catalog.

Delta Live Tables (Lakeflow Declarative Pipelines) define streaming tables with CREATE OR REFRESH STREAMING TABLE and materialized views with CREATE OR REFRESH MATERIALIZED VIEW, plus APPLY CHANGES for CDC.

Window functions such as ROW_NUMBER, RANK, LAG, LEAD, and SUM OVER require an OVER clause with PARTITION BY and ORDER BY; use them to deduplicate and compute running totals.

Data quality constraints use EXPECT ... ON VIOLATION DROP ROW or FAIL UPDATE; expectations are declared in pipeline code and reported in pipeline event logs.

Liquid clustering replaces partitioning and Z-ORDER for many tables; CLUSTER BY defines clustering keys that can be changed without rewriting data.

Change Data Capture with APPLY CHANGES INTO handles inserts, updates, and deletes from a change feed using SEQUENCE BY and KEYS clauses.

A Lakeflow job is a collection of tasks with dependencies; task types include notebook, Python script, SQL, dbt, pipeline, and run-job tasks, and tasks can run in parallel when they have no dependency.

Trigger types: scheduled (cron with timezone), continuous, file arrival, table update, and manual; a job can have multiple triggers but only one active run per trigger at a time.

Job parameters are passed to tasks via job parameters and task parameters; dynamic value references such as {{job.id}}, {{task.name}}, and {{job.start_time.iso_date}} are resolved at runtime.

Repair run reruns only failed or skipped tasks and their downstream dependencies; a full run reruns all tasks; both preserve the original run history.

Task dependencies form a DAG; each task runs on its own compute (job cluster, existing cluster, SQL warehouse, or serverless) and can have retries, timeouts, and notifications.

Notifications can be configured per job or per task for on-start, on-success, and on-failure events, delivered via email, webhook, or the Jobs API.

Job compute: serverless compute is available for notebooks and SQL tasks; job clusters terminate after the run; cluster reuse within a job is configured with the same cluster definition.

Databricks Repos (Git folders) link a workspace folder to a Git provider; supported operations include commit, push, pull, branch checkout, and create pull request.

Declarative Automation Bundles (formerly Databricks Asset Bundles) define jobs, pipelines, notebooks, and cluster configs in databricks.yml plus resource YAML files; validate with databricks bundle validate and deploy with databricks bundle deploy.

Bundle targets map to environments (dev, staging, prod) with per-target workspace hosts and variables; use databricks bundle deploy -t to select a target.

The Databricks CLI (v0.2xx and later) authenticates with a personal access token, OAuth, or a service principal; databricks auth login configures a profile.

CI/CD pipeline stages: build and unit test, validate bundle, deploy to dev, run integration tests, promote to prod; use service principals for automated deployments rather than personal tokens.

Terraform provider databricks is an alternative to bundles for provisioning workspaces, clusters, jobs, and Unity Catalog objects; state should be stored remotely.

Secrets for CI/CD should be stored in the CI system (GitHub Actions secrets, Azure DevOps variable groups) and referenced as environment variables, never committed to the repo.

Spark UI tabs: Jobs, Stages, Tasks, SQL, Storage, and Executors; the Stages tab shows shuffle read/write, spill, and task skew, which indicate skew or insufficient memory.

Common failure causes: OOM on the driver from collect() or large broadcast, OOM on executors from skewed partitions, and shuffle failures from disk space or lost executors.

Skew mitigation: salting keys, AQE skew join handling (spark.sql.adaptive.skewJoin.enabled), broadcast joins for small tables (spark.sql.autoBroadcastJoinThreshold), and repartitioning.

Adaptive Query Execution (AQE) coalesces shuffle partitions, converts sort-merge joins to broadcast joins, and handles skew; enabled by default in recent DBR versions.

Diagnostic logs: cluster log delivery to a cloud storage path, driver logs, and the Spark event log; pipeline event logs capture data quality metrics and lineage.

Query profiling: the Query Profile in the SQL editor shows operator-level time, rows, and spill; look for long scans, exploding joins, and excessive shuffles.

Delta optimization: OPTIMIZE with Z-ORDER or liquid clustering for file layout, VACUUM to reclaim storage, and ANALYZE TABLE to update statistics for the optimizer.

Monitoring: system tables (system.billing.usage, system.compute.clusters, system.access.audit) in Unity Catalog provide account-level observability; job run history and alerts are in the Jobs UI.

Unity Catalog hierarchy: metastore > catalog > schema > table/view/volume/function; privileges are granted at any level and inherited downward; USE CATALOG and USE SCHEMA are required to access objects.

Privilege types include SELECT, MODIFY, CREATE, USAGE, ALL PRIVILEGES, and ownership; owners and metastore admins can grant privileges; GRANT and REVOKE are the SQL commands.

Data isolation: dynamic views and row filter and column mask functions restrict access at query time; these are defined on tables and applied per user or group.

External locations require a storage credential backed by an IAM role or managed identity; external tables and external volumes must be under an external location.

Authentication: personal access tokens, OAuth machine-to-machine with service principals, and SSO via the identity provider; SCIM provisioning syncs users and groups from the IdP.

Audit logs record workspace and account activity; system.access.audit in Unity Catalog exposes audit events for querying; cluster log delivery and diagnostic logs support compliance.

Secrets are stored in secret scopes (Databricks-backed or Azure Key Vault-backed) and accessed with dbutils.secrets.get; secret values are redacted in notebook output.

Table ACLs and Unity Catalog grants differ: Unity Catalog is the recommended model and provides centralized governance across workspaces in an account.