Data Integration and Ingestion into Databricks
What this domain tests
Data Integration and Ingestion into Databricks tests how you move data efficiently: mapping design, load method, partitioning, write mode, and change capture. Questions often hinge on volume and latency.
Core rules
- Choose the load path by volume and shape. Small or narrow extracts can use the Databricks connector; large bulk loads often stage Parquet or Delta files in cloud storage and load with COPY INTO or a Databricks job. JDBC row-by-row reads are a last resort.
- Parallelize by partition key and size the agent or serverless runtime to match. Too few partitions underuse the cluster; too many create small files and commit overhead.
- Write modes matter: append adds rows, overwrite replaces a table or partition, and merge/upsert requires a key and Delta merge logic. Use merge for CDC and late-arriving updates.
- Mass ingestion and replication tasks suit many-source and CDC scenarios; mapping tasks suit curated transformations. Schema drift must be handled explicitly, not ignored.
Worked example
A 1 TB Oracle table must be copied nightly. A single JDBC session would be slow and fragile. Instead, extract in key-range partitions to Parquet in a cloud staging bucket, then run a Databricks job to validate and MERGE into a Delta bronze table. IDMC schedules and monitors the extract; Databricks owns the final write and can retry idempotently.
Traps
Assuming overwrite is safe for incremental data; creating thousands of small files; ignoring Delta commit conflicts from concurrent writers; and letting schema drift silently null out new columns.