Databricks Data Engineer flashcards
Databricks Data Engineer · Build recall with a focused review session.
32 cards. Read the term and the definition here. The deck at the top of this page is optional.
- Unity Catalog three-level namespace
- Catalogs contain schemas; schemas contain tables, views, volumes, functions, and models. Queries reference objects as catalog.schema.object, and privileges are granted at each level.
- Control plane vs data plane
- The Databricks control plane manages notebooks, jobs, and cluster configurations. The data plane is the cloud storage and compute in your own account where table data lives.
- Cluster access modes
- Shared mode allows multiple users with Python and SQL but restricts Scala and RDD APIs. Single-user mode gives one principal full language support; custom mode offers flexible isolation options.
- Photon engine
- A vectorized C++ execution engine available on Databricks clusters and SQL warehouses that accelerates SQL and DataFrame workloads against Delta tables.
- Delta Lake
- An open table format that adds ACID transactions, schema enforcement, time travel, and efficient upserts to Parquet files stored in cloud object storage.
- COPY INTO idempotency
- COPY INTO records the files it has already loaded and skips them on subsequent runs, so re-running the same statement over a directory does not duplicate rows.
- Auto Loader cloudFiles
- Incremental file ingestion that detects new files via directory listing or cloud file notifications. It needs a schema location for tracking and a checkpoint directory for stream state.
- availableNow trigger
- An Auto Loader trigger option that processes every file new since the last checkpoint in a single micro-batch and then stops, suitable for scheduled batch ingestion.
- Lakehouse Federation
- Unity Catalog foreign catalogs that let you query external databases such as PostgreSQL, MySQL, SQL Server, or Snowflake in place without copying data into Delta tables.
- MERGE INTO upsert
- A Delta Lake statement that matches source rows to target rows on a join key and performs updates, inserts, or deletes atomically in one transaction.
- Medallion architecture
- A layered data design where bronze stores raw ingested data, silver stores cleaned and conformed tables, and gold stores business-level aggregates used for reporting.
- ZORDER BY
- A Delta OPTIMIZE clause that colocates rows with similar values in the specified columns into the same files, improving data skipping for filtered queries.
- Liquid clustering
- A Delta table layout defined with CLUSTER BY columns that replaces fixed partitioning and ZORDER; clustering keys can be changed later without rewriting existing data.
- AUTO CDC APPLY CHANGES
- A Lakeflow pipeline operation that applies a change data capture feed to a target table, supporting SCD type 1 upserts and type 2 history with a sequencing column.
- Expectations in Lakeflow
- Data quality constraints declared on pipeline datasets using expect, expect_or_drop, expect_or_fail, or expect_all_or_quarantine; metrics are recorded on each pipeline update.
- run_if ALL_DONE
- A task dependency condition that runs the downstream task after all upstream tasks finish, whether those upstream tasks succeeded or failed.
- Repair run
- A job run option that re-executes only failed or skipped tasks, reusing the original run's parameters and the outputs of tasks that already succeeded.
- For each task
- A job task type that runs a nested task once per element of a JSON array input, with a configurable maximum concurrency.
- Job cluster
- An ephemeral cluster created for a single job run and terminated when it finishes; typically cheaper than an all-purpose cluster for scheduled workloads.
- Dynamic value reference
- Templated job syntax such as {{job.parameters.name}} or {{tasks.task.values.key}} that Databricks resolves at runtime from job or task context.
- Databricks Asset Bundle
- A declarative YAML project (databricks.yml) that defines jobs, pipelines, and permissions so the same resources deploy consistently to dev, staging, and prod targets.
- bundle deploy command
- The CLI command databricks bundle deploy -t <target> that validates the bundle and uploads its resources to the workspace defined by the selected target.
- OAuth machine-to-machine
- Service principal authentication using a client ID and secret to obtain OAuth tokens for automated pipelines, avoiding personal access tokens tied to an individual.
- Secret scope
- A named collection of secrets backed by Databricks or Azure Key Vault, read in code with dbutils.secrets.get(scope, key) so credentials never appear in notebooks.
- Adaptive Query Execution
- A Spark setting that re-optimizes shuffle partition counts, join strategies, and skew handling at runtime using statistics gathered after each shuffle stage.
- Task skew symptom
- In the Spark UI, one task's duration or shuffle read far exceeds the median, indicating uneven key distribution that may need salting or repartitioning.
- Broadcast join hint
- A SQL hint, /*+ BROADCAST(table) */, that replicates a small table to every executor so the join avoids shuffling the large table.
- Delta auto compaction
- A Delta table property, delta.autoOptimize.optimizeWrite, that coalesces small files written during a transaction to reduce small-file overhead.
- Row filter
- A Unity Catalog function attached to a table that returns a boolean per row, restricting which rows each querying user is allowed to see.
- Column mask
- A Unity Catalog function attached to a column that transforms values at query time, for example hashing, nulling, or partially showing sensitive data.
- External location
- A Unity Catalog object that pairs a cloud storage path with a storage credential, enabling creation of external tables and volumes at that path.
- system.access.audit
- A system table recording Unity Catalog and workspace audit events, including who accessed which object or ran which query and when.