Azure Databricks Data Engineer Associate common mistakes
Azure Databricks Data Engineer Associate · Understand the traps and how to avoid them.
Treating the workspace as the unit of data isolation and putting each team in its own workspace with its own metastore, then hand-copying tables between them.
Candidates carry over habits from relational systems where a server equals an environment. On Databricks the metastore is attached at the account level, so workspaces are compute and UI boundaries, not data boundaries. This shows up because the exam asks where catalogs and external locations live, and candidates answer with workspace-level constructs.
What to do instead
Learn the hierarchy: account, metastore, catalog, schema, table. Isolation between teams or environments is done with catalogs and grants inside one metastore, not with separate workspaces. Use one metastore per region and share it across workspaces, including dev and prod, then separate with catalog-level privileges and workspace bindings.
Using Auto Loader with a directory listing source (cloudFiles.useNotifications = false) for a high-volume landing zone and then wondering why ingestion cost and latency grow without bound.
Directory listing is the default and is the easiest thing to demo, so candidates assume it is the recommended production setting. It re-lists the whole prefix on each trigger, which is slow and expensive once the bucket holds millions of files. The exam tests whether you know when file notification mode is required.
What to do instead
For production volumes, enable file notification mode so Auto Loader uses cloud queue notifications plus a backfill for files that predate the queue. Reserve directory listing for small or one-off loads. Also set a schema location and a checkpoint so restarts are incremental, and use the cloudFiles backfill interval to catch files that arrive out of order.
Writing MERGE INTO without a deduplication step on the source, then hitting 'cannot perform merge because multiple source rows matched the target row' or silently producing the wrong final state.
Candidates test MERGE on clean sample data where the source key is unique. Real change feeds contain multiple updates per key in one batch. The exam presents a source with duplicate keys and asks what happens, and candidates assume MERGE picks the latest row automatically.
What to do instead
Deduplicate the source before the merge, typically with a window function over the key ordered by the change timestamp descending and row_number = 1, or use the source dedup helpers in the declarative pipeline APPLY CHANGES API. Know the difference between the error (multiple matches) and the silent case where a nondeterministic row wins.
Building a medallion pipeline as three independent notebooks chained by tasks that each re-read the previous table from storage, instead of using a declarative pipeline with streaming tables and materialized views.
The notebook-and-task pattern is what most people learn first, so it feels natural. Candidates then miss questions about how the declarative pipeline engine decides what to recompute, how it handles incremental refresh, and how expectations are declared.
What to do instead
Learn the declarative pipeline model: streaming tables for append-only incremental ingestion, materialized views for aggregations and transformations that need recomputation, and expectations (CONSTRAINT ... EXPECT) for data quality. Understand that the engine tracks dependencies and refreshes only what is stale, so you do not hand-orchestrate the layers.
Configuring a Lakeflow Jobs schedule with no retry policy and no notification on failure, and treating the job's success as proof the data is correct.
Candidates focus on getting the job to run once and never think about the operational envelope: retries, timeouts, alerts, and what happens when an upstream file is late. The exam asks how to make a pipeline resilient to transient cloud failures and how to alert a team when a run fails.
What to do instead
Set max retries with a retry interval for transient failures, set a timeout, and configure notifications for failure and for long-running runs. Use job parameters and task values to pass state between tasks rather than re-deriving it. Know that a successful run only means the code completed, so pair it with data quality checks (expectations or a validation task) that fail the job on bad data.
Putting notebooks and job definitions in Git but deploying them by clicking through the workspace UI, so the repository is never the source of truth.
The exam tests CI/CD concepts and candidates answer with what they do day to day, which is manual deployment. They also confuse Git folders (interactive editing) with the Databricks CLI or Terraform (automated deployment).
What to do instead
Separate the two concerns. Use Git folders for interactive development and branching. Use the Databricks CLI, the REST API, or Terraform in a CI pipeline to deploy bundles, jobs, and pipelines to each environment. Store job and pipeline definitions as code (declarative automation bundles or Terraform), and promote the same artifact across dev, staging, and prod rather than rebuilding per environment.
Diagnosing a slow query by adding more workers to the cluster, when the real problem is a skewed join key or a missing partition filter causing a full table scan.
Adding hardware is the intuitive fix and it sometimes helps a little, which reinforces the habit. The exam gives a query plan or a symptom description and expects you to identify skew, spill, or excessive scanning rather than scale out.
What to do instead
Read the query profile: look for skewed partitions (one task much larger than the rest), spill to disk, and scans that read far more data than the result. Fix skew with salting, broadcast of the small side, or skew hints. Fix scanning with partition and Z-order or liquid clustering on the filter columns, and with predicate pushdown. Only scale after the plan is sound.
Assuming a table's files are automatically compacted and that deleted rows disappear immediately, then being surprised by slow reads and stale results.
Delta Lake is often presented as 'just a table', so candidates do not internalize that small files accumulate from streaming writes and that deletes are logical until compaction. The exam asks about OPTIMIZE, VACUUM, and the retention window.
What to do instead
Know the maintenance operations and when to run them: OPTIMIZE to compact small files (with Z-order or liquid clustering for data skipping), VACUUM to physically remove files no longer referenced, and the default retention that protects time travel. Understand that VACUUM can break readers and time travel for periods shorter than the retention, so schedule it deliberately. For streaming sinks, enable automatic compaction or schedule OPTIMIZE.
Granting broad privileges at the metastore or catalog level to get a pipeline working, rather than granting the minimum at schema or table level and using groups.
Granting to a user or to 'all users' at a high level is the fastest way past a permission error, so it becomes the habit. The exam tests least privilege, group-based grants, and the interaction between Unity Catalog privileges and workspace-level cluster access.
What to do instead
Grant to groups, not individuals, and at the narrowest object that works: USE CATALOG and USE SCHEMA to traverse, SELECT or MODIFY on the table. Use external locations and storage credentials so data access is mediated by Unity Catalog rather than cloud IAM keys on the cluster. Know that a user needs both the Unity Catalog privilege and access to compute, and that cluster access modes (shared vs single-user) change what is allowed.
Confusing dynamic views and row filters with table ACLs, and expecting a column mask to also hide the column from a SELECT *.
The governance features sound similar and candidates treat them as interchangeable. In practice a column mask rewrites values on read but the column still appears, while a dynamic view can omit columns entirely. The exam asks which mechanism to use to hide a column versus mask its values.
What to do instead
Map the requirement to the feature. Use column masks to replace values for unauthorized users while keeping the column. Use row filters to restrict rows by predicate. Use dynamic views when the projection itself must change per user. All three are defined on the table or view and evaluated per query, so test with a low-privilege user rather than assuming the definition behaves as intended.