Develop Data Processing
Core concepts
Data processing in DP-203 spans Azure Data Factory (ADF), Synapse Pipelines, Databricks, and Stream Analytics. You must know how to design copy activities, data flows, and mapping data flows with transformations. Understand tumbling window vs hopping window triggers, and event-based triggers. For code-based processing, know PySpark DataFrame operations, Delta Lake ACID transactions, and merge/upsert patterns. Also cover Azure Functions for lightweight processing and HDInsight for legacy workloads.
Traps
A common trap is confusing data flow (visual, scales via Spark) with copy activity (no transformations). Another is trigger types: tumbling window preserves state and is best for recurring batch; event-based triggers fire on blob creation but can miss files if not idempotent. Also watch for polybase vs copy activity staging and self-hosted integration runtime requirements for on-premises sources.
Worked example
You need to incrementally load new files from ADLS Gen2 into a Delta table and upsert changes. Use a tumbling window trigger with a mapping data flow that includes an Alter Row transformation and a Delta sink with upsert. Alternatively, use Databricks with MERGE INTO for finer control.
Key rule
Choose the simplest tool that meets latency and transformation needs: copy activity for movement, data flows for visual transforms, Databricks notebooks for complex logic, Stream Analytics for real-time.