Databricks Lakehouse Concepts and Integration Patterns
What this domain tests
This domain tests lakehouse vocabulary, Databricks compute choices, and the integration patterns that connect IDMC to Databricks. You should be able to choose ELT versus ETL, batch versus CDC, and connector versus staging.
Core rules
- The lakehouse combines data lake storage with warehouse-style management. Delta Lake adds ACID transactions, schema enforcement, and time travel; the medallion layers are bronze (raw), silver (conformed), and gold (curated).
- Databricks compute includes all-purpose clusters for interactive work, job clusters for scheduled workloads, SQL warehouses for BI, and serverless options. Unity Catalog governs catalogs, schemas, tables, and privileges with a three-level namespace.
- Common IDMC-to-Databricks patterns: IDMC extracts and loads, Databricks transforms (ELT); IDMC transforms before load (ETL); IDMC pushes compatible transformations down to Spark; IDMC stages files in cloud storage for bulk loads; CDC replicates changes; reverse ETL publishes gold data to operational systems.
- The CDI Databricks connector can use a cluster or SQL warehouse endpoint depending on the load pattern; authentication uses a token or OAuth service principal. Verify the exact connector options in your org.
Worked example
A SaaS CRM must land in bronze every 15 minutes, then be modeled into gold by Databricks SQL. Use IDMC with a serverless runtime for the public SaaS API and a Databricks connection for the Delta write. Keep transformations in Databricks so the gold logic is versioned with the lakehouse code; IDMC owns extraction, scheduling, and load metadata.
Traps
Assuming IDMC automatically pushes every transformation to Databricks; using a SQL warehouse for large Spark-style writes without checking limits; and forgetting that Unity Catalog names need catalog.schema.table.