Fabric DP-700: Ingest and transform data

The ingestion toolbox

Match the tool to the shape of the problem:

Copy activity / Copy job in a pipeline - bulk movement, best for large one-shot or scheduled loads; supports staging for warehouse loads.
Dataflow Gen2 - Power Query Online transformations with a destination (lakehouse, warehouse, KQL DB, Azure SQL); low-code, good for moderate volumes.
Shortcuts - zero-copy access; nothing lands in your storage.
Mirroring - continuous, near-real-time replication of databases (SQL Server, Azure SQL, Cosmos DB, Snowflake) into OneLake as Delta.
Eventstream - streaming ingestion into an eventhouse or lakehouse.
Notebooks (PySpark) - complex, code-first transformations; MERGE INTO for upserts into Delta.

Medallion pattern

Bronze (raw append), Silver (cleansed, deduplicated, conformed), Gold (business-level aggregates serving semantic models). Delta Lake gives ACID transactions, time travel, schema enforcement/evolution, and MERGE.

Pipelines vs dataflows

Pipelines orchestrate (activities, parameters, triggers: schedule, tumbling window, event). Dataflows do the transformation logic. A pipeline can invoke a dataflow and chain notebooks.

Traps

Believing a shortcut copies data - it does not, and external shortcuts still need credentials/permissions at query time.
Expecting to write to a mirrored database - it is read-only.
Using Dataflow Gen2 for multi-terabyte transforms when Spark is the better fit.
Forgetting that Delta MERGE needs a stable key, and that schema evolution must be enabled for new columns.

Rule of thumb

"If you can avoid moving the data, shortcut or mirror it. If a copy must exist as Delta, use Copy activity. If the logic is complex or large-scale, use a notebook."