Spark Core and Spark SQL Development
Execution model
Spark RDDs are immutable and lazily evaluated; transformations build a lineage, actions trigger jobs. Narrow transformations keep partitions; wide transformations such as groupByKey, join, and repartition cause a shuffle. Spark SQL uses Catalyst for optimization and Tungsten for code generation and memory management. AQE can coalesce shuffle partitions and split skewed partitions.
Tuning rules
- Prefer reduceByKey or aggregateByKey over groupByKey to aggregate before shuffle.
- Broadcast a small dimension table; do not broadcast a table that exceeds executor and driver memory.
- Cache only reused DataFrames or RDDs and unpersist when finished.
- Set shuffle partitions from data size, not a fixed default.
- Use spark.sql.adaptive.skewJoin for known skew.
Worked example
A 1 TB sales fact table joins a 10 MB product dimension. A broadcast hash join sends the dimension to every executor and avoids shuffling the fact table. If the dimension were 10 GB, broadcast would cause OOM; use a sort-merge join with AQE and enough shuffle partitions instead.
Traps that cost marks
collect() on a large result pulls data to the driver and can OOM it. Caching every DataFrame wastes memory. A single hot key still bottlenecks one task after a normal join. Executor memory must include overhead; setting heap too high without overhead can kill containers.