I85PREP
⌘ K
Azure Databricks Data Engineer Associate study materials
Azure Databricks Data Engineer Associate study resources

Azure Databricks Data Engineer Associate: Data Ingestion and Loading

Azure Databricks Data Engineer Associate · Work through your exam, one topic at a time.

All guide sections

Data Ingestion and Loading

Data Ingestion and Loading

This domain tests how you land batch and streaming data into Delta tables on Azure Databricks. Questions focus on Auto Loader, COPY INTO, Kafka/Event Hubs, JDBC reads, schema handling, and idempotent incremental loads.

Core ingestion rules

  • Auto Loader (cloudFiles) is the scalable incremental file source. It tracks discovered files in a checkpoint, supports directory listing or file notification (Azure Event Grid/Queue), infers and evolves schema, and captures malformed records in _rescued_data. It is for files, not Kafka.
  • COPY INTO is a SQL command that loads new files idempotently by tracking file metadata in the Delta table. It suits SQL-centric, lower-volume incremental loads and supports mergeSchema and format options.
  • Kafka/Event Hubs: Azure Event Hubs exposes a Kafka-compatible endpoint, so the Kafka source works with Event Hubs connection properties. A checkpoint plus a Delta sink gives exactly-once processing; without a checkpoint you get duplicates or data loss.
  • JDBC: parallel reads need partitionColumn, lowerBound, upperBound, and numPartitions; otherwise a single task pulls the whole table. Store credentials in secret scopes or Key Vault-backed scopes.
  • Formats: Parquet, Delta, CSV, JSON, Avro, ORC, text. Specify schema hints to avoid expensive inference on large CSV/JSON backlogs.

Worked example

A landing container receives thousands of small CSV files per hour; a second source is a 10-row reference table in Azure SQL. Reasoning: use Auto Loader with file notification mode, a schema location, and mergeSchema for the CSV stream; the checkpoint gives incremental exactly-once semantics. For the tiny reference table, a JDBC batch read is fine, but broadcast it before joining. COPY INTO would be simpler only if the team were SQL-only and file volume were low.

Traps

  • Using COPY INTO for continuous high-volume ingestion.
  • Forgetting the checkpoint on a streaming read, or reusing one checkpoint for two queries.
  • Assuming Auto Loader reads Kafka; it does not.
  • Letting schema inference run on every batch instead of persisting a schema location.
  • Reading from DBFS root or an ungoverned path when UC external locations are required.