Databricks Data Engineer study plan

Configure Auto Loader to ingest JSON and CSV files from cloud storage into a bronze table with schema inference and schema evolution enabled

Implement a COPY INTO batch load with idempotent re-runs and validate that rerunning the same files produces no duplicate rows

Capture malformed records using the rescued data column and route them to a quarantine table

Choose between managed and external Unity Catalog tables for landing vs. curated layers and justify the choice in writing

Build a streaming pipeline that reads bronze change data and applies CDC merges into a silver table using AUTO CDC or APPLY CHANGES

Add data-quality expectations to the silver layer and demonstrate what happens on warn vs. drop vs. fail

Handle late-arriving and out-of-order records with watermarking and sequence columns

Parameterize the pipeline so the same code targets dev and prod catalogs without edits

Deduplicate source records using window functions or dropDuplicates and document the tie-breaking rule used

Model a gold star schema with one fact table and at least two conformed dimensions at a declared grain

Implement SCD Type 2 history for one dimension and query it as-of a past date

Lay out the medallion layers with clear naming, ownership, and column-comment conventions

Read EXPLAIN and query profiles to identify shuffles, skewed joins, and full scans in your pipelines

Apply partition pruning, file compaction, and clustering keys to the largest tables you built

Replace an expensive join or UDF with a broadcast join or built-in function and measure the effect

Set up a repeatable benchmark harness that times the same query before and after each change

Author a multi-task job with explicit task dependencies, shared job parameters, and per-task cluster or serverless compute choices

Configure retries, timeouts, and email or webhook notifications on failure

Schedule the job with a trigger and use run repair to re-run only failed tasks after a fix

Pass values between tasks using task values and verify them in downstream results

Grant catalog, schema, and table privileges to two distinct groups and verify least-privilege access by testing both identities

Implement a column mask and a row filter on a PII table and confirm a restricted user sees only permitted data

Create a storage credential and external location, then restrict where external tables can be written

Review audit logs and lineage for one table to confirm who read it and where its data came from

Package pipelines and jobs as a Databricks Asset Bundle with dev and prod targets

Store environment-specific values in bundle variables and secrets rather than hard-coded strings

Run bundle validation and deployment from a CI pipeline triggered on pull request

Add a lightweight data test that runs in CI and fails the build on broken transformations

Compare serverless, job cluster, and SQL warehouse compute options and pick the right fit for three given workloads

Size clusters using the metrics you collected and identify idle or over-provisioned resources

Use system tables and query history to find the slowest and most expensive statements from weeks 1-7

Complete a timed full-length practice exam and log every missed question by domain

Write a remediation list that re-drills only the two weakest domains before test day