Databricks Spark Developer Associate: Spark Architecture and Components

Cluster anatomy

An application is a driver (runs main(), owns the SparkSession/SparkContext, builds the DAG, schedules work, collects results) plus a set of executors (JVM processes that run tasks in core slots and hold cached blocks). A cluster manager - Standalone, YARN, Kubernetes, or Databricks' managed layer - allocates executors. Driver memory is separate from executor memory; collect(), broadcast variables and toPandas() all land on the driver, which is the classic OOM source.

Jobs, stages, tasks

An action creates a job; the DAG scheduler cuts it into stages at wide transformations (groupBy/agg, non-broadcast join, distinct, repartition, orderBy) because each needs a shuffle. Every stage has one task per partition, so partition count - not executor count - predicts parallelism.

Traps

The exam loves these:

• Nothing runs until an action. select, filter, withColumn, join are lazy; count, show, collect, write are not. cache() also materialises only on the first action.
• Narrow transformations (select, filter, withColumn, same-width union) pipeline inside one stage and never shuffle.
• Adding executors cannot speed up a stage that only has four partitions.
• spark.default.parallelism governs RDD/shuffle default parallelism, while spark.sql.shuffle.partitions (default 200) governs SQL/DataFrame shuffles - they are different knobs.

Rule of thumb

Target 100–200 MB per partition and about 2–4 concurrent tasks per core. With 20 executors × 4 cores = 80 slots, 200 shuffle partitions wastes slots on tiny tasks, while a few thousand balanced partitions keeps all cores fed without spilling.