Databricks Spark Developer Associate: Troubleshooting and Tuning DataFrame Applications

Diagnose before tuning

Read .explain(True) for the parsed, analysed, optimised and physical plans. Look for Exchange (shuffle), BroadcastHashJoin vs SortMergeJoin, FileScan with pushed filters, and AdaptiveSparkPlan. The Spark UI shows spill (memory/disk), skewed task durations, GC time, and shuffle read/write volume.

Levers that matter

• Partition sizing: spark.sql.files.maxPartitionBytes (128 MB default) and spark.sql.shuffle.partitions. AQE (spark.sql.adaptive.enabled, on by default in modern runtimes) coalesces small shuffle partitions, splits skewed ones, and switches sort-merge to broadcast join at runtime.
• Joins: spark.sql.autoBroadcastJoinThreshold defaults to 10 MB. Broadcasting a small dimension eliminates the shuffle; F.broadcast(df) forces it.
• Caching: persist(MEMORY_AND_DISK) only pays off if a DataFrame is reused across multiple actions; otherwise it steals memory from execution.
• Skew: one task running 10× longer than the rest signals a hot key. Fixes are salting the key, splitting the skewed rows out and unioning back, or letting AQE handle skew joins.
• Small files: compact with repartition(n) (full shuffle, can increase or decrease) or coalesce(n) (no shuffle, can only decrease). Delta OPTIMIZE compacts on disk.

Traps

• coalesce cannot increase partitions and gives up upstream parallelism.
• Raising spark.sql.shuffle.partitions to 4000 does not fix skew - it makes many empty tasks.
• Confusing driver OOM (collect) with executor OOM (partition too large / skewed key).
• Adding repartition(1) "to make it faster" - it funnels everything through one task.

Rule of thumb

If tasks spill heavily, reduce partition size; if tasks are tiny and the scheduler overhead dominates, coalesce. Aim for tasks of roughly one minute.