Databricks Data Engineer Professional study plan
Databricks Data Engineer Professional · Give each week a clear focus.
8 weeks · About 7 hours per week
Developing Code for Data Processing using Python and SQL
Write PySpark DataFrame transformations using select, withColumn, join, and Window functions without relying on references
Implement schema-defined reads and writes across Parquet, Delta, and JSON with explicit options
Author equivalent SQL and Python implementations of the same aggregation and return identical row counts
Register and call Python UDFs and explain when to prefer built-in or higher-order functions instead
Your checkpoint
A notebook containing four transformation tasks, each solved twice (PySpark and SQL), with matching output row counts recorded in a summary table
Developing Code for Data Processing using Python and SQL & Debugging and Deploying
Build a Structured Streaming pipeline in Python that reads a source incrementally and writes to a Delta sink with checkpointing
Package job code into a Python wheel or notebook bundle and parameterise it with widgets or job parameters
Reproduce and fix three seeded Spark failures by reading the Spark UI and driver logs
Unit test transformation functions with a local Spark session and assert on schema and content
Your checkpoint
A parameterised streaming job plus a passing local test suite whose failure log analysis for three seeded bugs is written up in a short troubleshooting note
Cost & Performance Optimisation & Monitoring and Alerting
Diagnose shuffle, skew, and spill in a job plan and apply at least two remediations such as broadcast joins or salting
Configure Delta table properties and layout choices (partitioning, Z-order or liquid clustering) for a given query pattern
Instrument a pipeline with metrics emitted to a log table and configure an alert on a threshold breach
Compare job cluster sizing and autoscaling settings and justify a cost trade-off in writing
Your checkpoint
A before-and-after benchmark report for one slow job showing runtime, shuffle read, and cluster cost per run, plus a working alert on a failure-rate metric
Data Transformation, Cleansing, and Quality & Data Ingestion & Acquisition
Implement ingestion from files and a message queue using Auto Loader with schema inference, evolution, and rescue handling
Encode data-quality expectations that quarantine or fail on bad records rather than dropping them silently
Deduplicate and reconcile late-arriving records in a transformation layer with documented semantics
Write idempotent transformations that produce identical results when re-executed on the same input
Your checkpoint
An ingestion-to-cleansed pipeline with a quarantine table, a re-run proof showing unchanged output counts, and a metrics snapshot of rejected records by reason
Ensuring Data Security and Compliance & Data Governance
Configure catalog, schema, and table-level privileges and verify them with a second principal
Apply column masks, row filters, and dynamic view-based access to a shared table and confirm the redaction
Set up tags, comments, and ownership conventions, then trace lineage for one derived table
Classify a table's data sensitivity and map each class to a concrete control
Your checkpoint
A security demonstration notebook showing allowed versus denied results for two principals, plus a one-page governance artefact listing table owners, tags, and sensitivity classes
Data Modelling & Data Sharing and Federation
Model a source system as a star schema and implement the fact and dimension tables with surrogate keys
Implement slowly changing dimension behaviour for at least one dimension and validate historical rows
Share a table with an external recipient and query a federated external source from the lakehouse
Document the grain, keys, and refresh strategy for each modelled table
Your checkpoint
A dimensional model with one SCD-tracked dimension and a sharing or federation proof, accompanied by a schema document stating grain and key definitions
Monitoring and Alerting & Debugging and Deploying
Simulate a production incident, isolate root cause from logs and lineage, and restore the pipeline
Write a runbook covering failure triage, restart strategy, and alert escalation for one pipeline
Harden a job for deployment by externalising config and adding a rollback path for a bad release
Add a data-freshness check that fires an alert when a table is not updated within its SLA
Your checkpoint
A runbook plus a timed incident drill record noting detection time, root cause, recovery action, and the alert evidence that fired
Developing Code for Data Processing using Python and SQL & Cost & Performance Optimisation
Complete two full-length timed practice exams and score every missed item against the ten domains
Re-implement the three weakest question areas from scratch without notes
Re-time the Week 3 benchmark job and confirm the optimisations still hold under a fresh run
Consolidate a one-page cheat sheet of decisions, defaults, and gotchas per domain
Your checkpoint
A score tracker across two full practice exams plus a one-page cheat sheet, with every domain scoring below 75 percent paired to a concrete remediation task
Milestones
Week 2: A streaming job deploys with parameters and its local test suite passes; three seeded failures are diagnosed with evidence from logs
Week 4: The end-to-end ingestion and cleansing pipeline re-runs idempotently, and the quarantine table accounts for every rejected record
Week 6: Two principals get different, verified results on a protected table, and the dimensional model's grain and keys are documented
Week 8: Two timed mocks are complete with per-domain scores recorded; no domain remains below 75 percent without a written remediation task