Databricks Data Engineer: Troubleshooting, Monitoring, and Optimization

Where to look

Spark UI / query profile: stages → tasks. Look for shuffle read/write volume, spill (memory versus disk), and task duration skew (max versus median). Delta history (DESCRIBE HISTORY) exposes operation metrics such as numFiles, numOutputRows, and whether OPTIMIZE ran. System tables (system.billing.usage, system.compute.*, system.access.audit, system.query.history) show cost and usage trends. Driver and cluster logs are delivered to a UC volume or DBFS destination you configure.

Optimization levers

• AQE (on by default) coalesces shuffle partitions, splits skewed partitions, and can switch join strategy.
• Broadcast join the small side via hint or autoBroadcastJoinThreshold to eliminate a shuffle.
• Skew hints for known hot keys that AQE cannot split.
• Cache only DataFrames reused many times; unpersist when finished.
• Compaction: OPTIMIZE for small files, liquid clustering instead of partitioning for new tables, and predictive optimization on UC managed tables.
• Photon accelerates native scans, joins, and aggregates, not Python UDFs - rewriting a Python UDF as native functions usually beats resizing the cluster.

Traps

• Reading spill as a disk failure; it signals executor memory pressure.
• Expecting AQE to fix non-shuffle skew from one giant input file.
• Assuming a larger cluster fixes bad join order or missing predicate pushdown.
• Running VACUUM with retention shorter than the longest reader's time-travel horizon.

Rule of thumb

When a stage is slow, compare slowest task to median. One dominating task means skew; all tasks slow with spill means memory or partition sizing; huge shuffle bytes means a join-strategy problem.