Databricks Data Engineer common mistakes
Databricks Data Engineer · Understand the traps and how to avoid them.
Selecting an all-purpose cluster (or a single-node cluster) as the compute for a scheduled Lakeflow Job run, or assuming a serverless SQL warehouse can execute Python/Spark code in a task.
Candidates learn on interactive all-purpose clusters and carry that habit into production job definitions; the exam scenarios describe scheduled pipelines where the compute choice affects isolation, autoscaling, and cost, and where SQL-warehouse-only tasks cannot run notebook/Spark code.
What to do instead
In job definitions use job compute (or serverless job compute) so each run gets isolated, right-sized resources; use a serverless SQL warehouse only for pure SQL tasks, and move Python/Spark logic to a notebook or Python wheel task on job compute.
Implementing incremental ingestion by re-reading the whole landing directory each run with a DataFrame read or CREATE TABLE AS SELECT, then overwriting the target.
It is the fastest pattern to write locally, and candidates do not distinguish 'the query returned new data' from 'the platform tracked which files were already consumed' - the ingestion state is implicit and lost between runs.
What to do instead
Use Auto Loader (cloudFiles) with a required checkpoint location, or COPY INTO for SQL/SQL-warehouse ingestion; both persist file-level ingestion state so re-runs are idempotent and only new files are processed.
Assuming that because Auto Loader inferred a new column in the source JSON, the downstream Delta table will automatically gain that column, and ignoring the rescued-data column when records arrive with unexpected shapes.
Schema inference on the stream and schema of the persisted sink table are two separate things, and malformed records land in a rescue column rather than failing loudly, so candidates believe data was loaded when it was quarantined.
What to do instead
Explicitly enable schema evolution on the write (e.g. mergeSchema) or use a controlled schema-evolution step, add cloudFiles.schemaLocation so the inferred schema is stable across restarts, and monitor/query _rescued_data to catch records that were not parsed.
Using MERGE INTO to upsert from a source that still contains duplicate keys, expecting the last row to win.
MERGE reads as 'SQL upsert' so candidates assume duplicate handling is implicit; in Delta, when more than one source row matches the same target row for an UPDATE or DELETE action the statement fails (DELTA_MULTIPLE_SOURCE_ROW_MATCHING_TARGET_ROW_IN_MERGE).
What to do instead
Deduplicate the source before the merge - a window function with ROW_NUMBER() over the key ordered by an update timestamp, keeping rn = 1 - or pre-aggregate to one row per key; never rely on statement ordering.
Defaulting to PARTITION BY on a low-cardinality or rarely filtered column (or partitioning a small table at all), instead of choosing clustering keys based on query predicates.
Partitioning is the classic Hadoop-era habit and looks like 'good modelling'; over-partitioning creates many tiny files, and partition columns are also immutable in place, which blocks later design changes.
What to do instead
Use liquid clustering (CLUSTER BY) on the high-cardinality columns that appear in filter/join predicates, leave small tables unpartitioned, and reserve classic partitioning for very large tables filtered on the same low-cardinality column; note liquid clustering and PARTITION BY are mutually exclusive.
Assuming 'Repair run' re-executes the entire job from the first task, or conversely that a retry on a single task will automatically re-run the downstream tasks that consumed its output.
Candidates conflate job-level reruns with task-level retries and do not reason about the task dependency graph declared via depends_on, so they mis-predict which tasks run and which are skipped.
What to do instead
Model the DAG explicitly with depends_on; use Repair run to re-execute only the failed tasks and their downstream dependents while leaving succeeded tasks untouched, and set task-level retries/timeouts for transient failures rather than rerunning the whole job.
Hardcoding workspace URLs, catalog/schema names, and cluster IDs directly into notebooks and job JSON, so the same artifact is deployed to every environment.
It works in a single dev workspace and candidates never introduce a parameterization layer, so promotion to staging/prod either fails or silently writes into the wrong catalog.
What to do instead
Package the project as a Databricks Asset Bundle with per-environment targets and ${var.*} variables (resolved from target-specific values or environment variables), validate with databricks bundle validate -t <target> and deploy with databricks bundle deploy -t <target> so no environment-specific literal lives in source.
Putting transformation logic directly inside notebook cells and then trying to satisfy the CI test gate by executing notebooks against a live cluster (or skipping tests because 'notebooks are untestable').
Notebook-only development is the default on the platform, and spinning up compute inside CI is slow and credential-heavy, so teams either accept flaky integration tests or no coverage at all.
What to do instead
Move transformation logic into importable Python modules/wheels that take DataFrames as input, unit-test them with a local Spark session and tiny fixtures in CI, and keep notebooks as thin orchestration wrappers that pass bundle-resolved parameters.
Diagnosing a failed/long-running Spark job from the top-level 'Job aborted due to stage failure' message or by enlarging the cluster, without reading the innermost Caused by and the stage/task metrics in the Spark UI.
The platform surfaces the outermost exception prominently, and adding compute is the intuitive first response; the actual root cause (data skew, OOM in a specific stage, a permissions error on a path) is buried in the cause chain or visible only per-stage.
What to do instead
Walk the full exception chain to the root cause, open the Spark UI / query profile for the failing stage, and check task duration distribution for skew and spill/memory metrics; then fix the cause (AQE skew join, repartition on a better key, broadcast join, file compaction) instead of scaling out.
Granting SELECT on a table to a group and assuming access works, and validating row filters/column masks by querying as the table owner.
Unity Catalog requires a privilege chain (USE CATALOG, USE SCHEMA, then the object privilege), and owners bypass row filters and column masks, so a test run as the owner always shows unfiltered data and the privilege gap only appears in production for other users.
What to do instead
Grant USE CATALOG on the catalog and USE SCHEMA on the schema in addition to SELECT on the table; verify filtering with a non-owner test identity (or the actual group member) and confirm which roles are exempt from the policy before declaring it correct.