Databricks Data Engineer Professional case studies
Databricks Data Engineer Professional · Put your knowledge to work in context.
The Databricks Certified Data Engineer Professional is a proctored certification exam that validates advanced skills in building, optimising, and maintaining production-grade data engineering solutions on the Databricks Data + AI Platform, a multi-cloud environment. It is aimed at practitioners who already have hands-on experience with the platform rather than at newcomers: the source lists no formal prerequisites but states that hands-on experience performing the data engineering tasks outlined in the exam guide is recommended, and that related training is highly recommended. The Databricks Data Engineer Professional case studies are 5 cases and about 3,894 words.
1The nightly Z-ORDER job that costs 4 hours and still scans 290 GB
Your team runs a clickstream pipeline that lands vendor JSON into bronze and appends to silver.events , a Delta table of 8 TB that grows by about 40 GB per day. The silver write is a Structured Streaming job with a 30-second trigger, so roughly 2,900 new files arrive every day.
792 words, about 4 min2Auto Loader feeding on a 40-million-object bucket with a changing vendor schema
An Auto Loader stream ingests vendor JSON from s3://acme-landing/orders/ into a bronze Delta table using directory listing mode. The bucket holds about 40 million objects and about 120,000 new 4 MB files land per day.
790 words, about 4 min3A MERGE that fails on duplicates, goes stale on retry, and silently keeps deleted orders
silver.orders is fed from bronze.orders cdc , an append-only stream of CDC events from Kafka (about 9 million rows a day, roughly 40 events per second across about 2 million order keys). A Structured Streaming foreachBatch job merges each micro-batch into silver.orders (SCD type 1, 300 million rows) with MERGE INTO silver.orders t USING batch b ON t.order id = b.order id WHEN MATCHED THEN UPDATE SET WHEN NOT MATCHED THEN INSERT .
737 words, about 4 min4Federated Postgres join hammering the OLTP primary, plus a partner share with PII
Analysts in your Unity Catalog metastore need to join prod.finance.invoices , a 600 GB Delta table, with ops.pg billing.customers , a Lakehouse Federation connection to the billing team's operational PostgreSQL instance (roughly 2,000 rows per region, 40 regions). A first query joined the federated table directly to a Delta subquery filtered to one month; it ran 47 minutes and pinned the Postgres primary, prompting an angry call from the billing on-call.
799 words, about 4 min5A stream that dies after VACUUM, with a bad deploy waiting to be rolled back
At 02:10 a nightly Structured Streaming job fails with A file referenced in the transaction log cannot be found . The stream reads silver.sessions and has been running for 11 days without a restart.
776 words, about 4 min