Data Ingestion & Acquisition
Batch file ingestion
COPY INTO and Auto Loader are the two managed paths into Delta. Both are idempotent by tracking which files were already processed, so a retried job does not duplicate rows. COPY INTO is SQL-first, best for thousands of files, and re-reading the source listing each run; it supports PATTERN, FORMAT_OPTIONS, and COPY_OPTIONS. Auto Loader (cloudFiles) scales to millions of files, keeps state in a checkpoint, offers availableNow (batch-like) and continuous triggers, and can use file notification mode (cloud queue + notification services) instead of directory listing to cut listing cost. Schema handling is the biggest trap: cloudFiles.schemaLocation stores an inferred schema that evolves with schemaEvolutionMode; a stale or deleted schema location causes unexpected types. rescuedDataColumn captures records that did not match the schema instead of dropping them - the exam likes asking where malformed rows went.
Streaming and database sources
Know that streaming ingestion needs a checkpoint location per query, and that changing the source or query logic can invalidate it. For Kafka, the value is binary - cast it; use startingOffsets, maxOffsetsPerTrigger, and trigger intervals to bound micro-batches. For JDBC, control read parallelism with numPartitions, partitionColumn, lowerBound, upperBound; those only work with an integer/date partition column.
Rule of thumb: landing zone → Auto Loader streaming table (bronze) → cleansed silver; never point a streaming query at a small file set that keeps growing without a checkpoint you control.