Azure Databricks Data Engineer Associate: Troubleshooting, Monitoring, and Optimization

Troubleshooting, Monitoring, and Optimization

This domain tests diagnosis and tuning of Spark and Delta workloads, plus the monitoring surfaces you use in production. Questions describe symptoms such as slow stages, spills, or small files and ask for the best fix.

Diagnosis and tuning rules

In the Spark UI, compare task durations: one long task among many indicates skew; high shuffle read/write indicates wide transformations; spill to disk indicates insufficient memory or oversized partitions; high GC time suggests memory pressure.
Adaptive Query Execution (AQE) coalesces shuffle partitions, handles skew joins, and applies dynamic partition pruning. Set a sensible spark.sql.shuffle.partitions; do not assume more workers fix skew.
For a small dimension table, use a broadcast hash join; for a large skewed key, use AQE skew join or salting. repartition causes a full shuffle; coalesce reduces partitions without one.
Delta optimization: OPTIMIZE compacts small files; ZORDER co-locates related data for skipping (legacy pattern); liquid clustering is the newer layout choice; VACUUM removes unreferenced files after the retention period (default 7 days). Predictive optimization can run maintenance automatically.
Monitoring: job run history and cluster event logs, query history and query profile for SQL warehouses, diagnostic logs exported to Azure Log Analytics, and Unity Catalog system tables for audit, billing, and job timelines.

Worked example

A job reads a 2 GB fact table and a 5 MB dimension, then aggregates by customer_id. One task runs for 40 minutes while others finish in 2 minutes. Reasoning: the dimension is small, so broadcast it and avoid the shuffle. The long task suggests skew on customer_id; enable AQE skew join or salt the hot key. Check spill in the stage detail; if present, increase executor memory or reduce partition size. Adding workers alone would leave the hot task unchanged.

Traps

Scaling the cluster when a single key is skewed.
Running VACUUM with retention below the Delta retention safety threshold.
Using ZORDER on a liquid-clustered table.
Caching every DataFrame and exhausting memory.
Ignoring small-file compaction after streaming ingestion.
Monitoring only cluster CPU and not job/query history.