I85PREP
⌘ K
Databricks Data Engineer study materials
Databricks Data Engineer study resources

Databricks Data Engineer: Data Transformation and Modeling

Databricks Data Engineer · Work through your exam, one topic at a time.

All guide sections

Data Transformation and Modeling

Medallion and Delta fundamentals

Bronze preserves raw data, silver holds conformed and deduplicated entities, gold holds business aggregates. Delta Lake provides ACID transactions, schema enforcement (with mergeSchema opt-in), time travel, MERGE, deletion vectors, and liquid clustering. Column statistics enable data skipping; OPTIMIZE compacts small files; Z-ORDER co-locates related values; VACUUM removes unreferenced files (default 7-day retention, and time travel past that horizon stops working).

Modeling moves

  • Deduplicate before merging using ROW_NUMBER() over the business key ordered by ingestion timestamp.
  • SCD Type 1 overwrites attributes; SCD Type 2 inserts a new version and closes the prior row with valid_to and is_current. In pipelines use APPLY CHANGES INTO with a sequencing column.
  • Partition only large tables (roughly >1 TB) on low-cardinality columns; prefer liquid clustering over partitioning plus Z-ORDER for new tables because clustering keys can change without rewriting history.
  • Never partition on high-cardinality columns - you get small files and slow listing.

Traps

  • MERGE erroring on duplicate source keys; fix the source, not the merge.
  • Using INSERT OVERWRITE where late-arriving data must be preserved.
  • Writing many tiny files from a streaming job without compaction.
  • Assuming VACUUM deletes rows from history - it deletes files, and older queries then fail.

Worked example

To keep a silver customers table current: stage the batch, dedupe by customer_id, then MERGE on customer_id with a matched-update branch setting updated_at = current_timestamp() and a not-matched insert branch, followed by a nightly OPTIMIZE.