Databricks Data Engineer Professional common mistakes
Databricks Data Engineer Professional · Understand the traps and how to avoid them.
Writes row-by-row transformations as Python UDFs (or uses .collect()/toPandas() to loop over rows) when the same logic is expressible with built-in functions, then wonders why the job that ran fine on 1 GB takes hours on 500 GB.
Python UDFs are the most familiar pattern for engineers coming from pandas/scripting backgrounds, and they are the first thing that works on a laptop-sized sample. The candidate never re-tests at production scale, so the serialization cost and the loss of Catalyst optimisation and predicate pushdown stay invisible during development.
What to do instead
Rewrite with pyspark.sql.functions (when/otherwise, explode, transform, filter, aggregate, higher-order functions). If a UDF is genuinely required, use an Arrow-optimised pandas UDF or a Scala/Java UDF, and compare the physical plan and runtime before and after. Treat .collect() on an unaggregated DataFrame as a defect in review.
Runs a streaming Auto Loader ingest without a dedicated checkpointLocation per stream, or copies a working notebook to a new stream and reuses the old checkpoint path, then reports 'the stream reads no new files' or sees duplicated rows.
The checkpoint is the only record of which files were already processed; it is easy to omit when developing interactively because the first run succeeds and produces data. Copy/paste of a proven notebook also copies its checkpoint literal.
What to do instead
Always set a unique cloudFiles.checkpointLocation and a separate cloudFiles.schemaLocation (for example derived from the pipeline ID and stream name), keep both outside the data path, and use trigger(availableNow=True) for incremental batch-style loads. When intentionally reprocessing, delete or version the checkpoint deliberately rather than silently sharing it.
Builds a MERGE INTO from a raw change feed without deduplicating the source, hitting 'Cannot perform MERGE as multiple source rows matched' or, worse, succeeding with non-deterministic final values when the merge condition is not fully deterministic.
The candidate tests on a sample where each key appears once; real change feeds contain multiple updates per key within the same micro-batch window. The error only surfaces under production volume and ordering.
What to do instead
Aggregate the source to exactly one row per merge key first (row_number() over partition by key order by ingestion/event timestamp desc, or max_by), then merge from that deduplicated subquery. Verify determinism by checking the merge condition matches a unique source key and confirm with a post-merge duplicate count.
Chooses Lakehouse Federation when the requirement is to let an external partner query curated tables, or chooses Delta Sharing when the requirement is to join a Delta table against a live PostgreSQL table without copying it, and defends the choice with 'both move data'.
Both features are described as 'sharing data' in overview material, and the candidate pattern-matches on the word 'share' rather than on direction of data flow (leaves the metastore vs queried in place) and on who initiates the access.
What to do instead
Decide from two questions: is the data crossing an organisational/account boundary and read-only to the recipient (Delta Sharing: share, recipient, activation link, credential file, read-only), or is a consumer inside your account querying an external system in place (Federation: connection, foreign catalog, pushdown of predicates)? Then check that downstream consumers only need SELECT on the share or foreign catalog, not on the storage.
Sets up monitoring that only alerts on job/pipeline run failure, so a pipeline that succeeds while dropping 40% of records into the rescued-data column, or whose row counts have collapsed, is never surfaced.
'Green run' feels like success, and the failure email is the first monitoring feature a candidate discovers. Silent quality regressions and partial ingests produce no failure status at all.
What to do instead
Add in-pipeline expectations (expect / expect_or_drop / expect_or_fail) on key columns and row counts, wire Databricks SQL alerts on a counts-and-freshness query over the target table, and use system tables (system.lakeflow job/pipeline history, system.billing) to alert on duration and cost anomalies as well as status.
Partitions a Delta table by a high-cardinality column such as a timestamp or an id 'to make queries fast', then reports slow reads and thousands of small files, or applies OPTIMIZE/Z-ORDER on a table whose access pattern is only full scans.
Partitioning is the best-known tuning knob and looks like a safe default; the candidate does not check the resulting file sizes or read the query profile to see whether pruning actually occurs.
What to do instead
Default to no partitioning and use liquid clustering on the columns actually used in filters, then OPTIMIZE to compact files (target roughly 100 MB–1 GB). Verify with DESCRIBE DETAIL (numFiles, sizeInBytes) and the query profile that files/partitions are pruned. Respect VACUUM retention when cleaning up so time travel is not silently broken.
Makes a row-filtering dynamic view work by granting users SELECT on the underlying base table, or applies a dynamic view where a column mask function is the correct control, effectively handing out the data the control was meant to restrict.
The view returns an error for the user until the base table is readable, and granting access is the fastest way to make the error go away. The candidate is optimising for 'query works' rather than for the security property being enforced.
What to do instead
Keep base-table privileges with the view owner (a group), grant consumers only SELECT on the view, and confirm with SHOW GRANTS and by querying as a low-privilege principal. Use dynamic views for row-level filters and column mask functions for column-level redaction; remember users still need USE CATALOG/USE SCHEMA to resolve the view.
Builds governance artefacts (impact analysis, PII inventory, 'who reads this table') by grepping notebook source or maintaining a hand-written spreadsheet of table relationships, instead of querying Unity Catalog lineage and audit data.
Candidates treat lineage as a diagram in the UI rather than as queryable data, and underestimate how quickly manually maintained inventories go stale once pipelines are added by other teams.
What to do instead
Query system.access.table_lineage and system.access.column_lineage for upstream/downstream impact, system.access.audit for access patterns, and information_schema for table/column metadata and tags. Combine with UC tags and comments applied at scale (ALTER TABLE ... SET TAGS) so the inventory is derived, not hand-maintained.
Hardcodes catalog names, storage paths, cluster sizes and secret values directly in notebooks, and deploys by clicking 'Run now' or copying notebooks between workspaces, so promotion to production requires manual edits that get missed.
Interactive development happens against one workspace with literal paths, and the notebook works immediately, so the deployment path is never exercised until a release. Secrets in code also pass review because 'it's only a dev token'.
What to do instead
Parameterise via job/pipeline parameters, widgets with defaults, and spark.conf, externalise secrets to secret scopes or secret managers (never literals), and deploy with Databricks Asset Bundles (or Terraform) through CI so the same artefacts run in every environment with only variable files differing. Keep environment-specific values in bundle targets, not in notebook code.
Defines an SCD Type 2 or 'latest state' pipeline with APPLY CHANGES INTO without SEQUENCE BY (or without delete handling), then finds the target row reflects an older update because events arrived out of order.
Kafka/CDC feeds are commonly late and unordered, but test data is usually written in order, so the first run looks correct. The candidate assumes the target table's own write order determines 'latest'.
What to do instead
Declare SEQUENCE BY on an event/monotonic column, set APPLY AS DELETE WHEN for soft deletes, and explicitly choose stored_as_scd_type 1 vs 2 plus the tracked columns. Validate by replaying a deliberately shuffled sample and asserting the target matches the max-sequence row.