Databricks Spark Developer Associate cheat sheet

Application anatomy: one driver process runs the SparkSession and DAG scheduler; the cluster manager allocates executor JVMs; each executor offers spark.executor.cores task slots and spark.executor.memory.

Action triggers a job; the job is cut into stages at every wide dependency (shuffle); each stage runs one task per partition, so tasks in a stage = partition count at that point.

Narrow dependency (map, filter, union, coalesce) keeps each parent partition feeding at most one child partition and needs no network transfer.

Wide dependency (groupBy, distinct, repartition, join on a non-broadcast key) makes one child partition depend on many parents and forces a shuffle plus a new stage.

Lazy evaluation: transformations only extend the lineage DAG; nothing runs until an action such as count(), collect(), show(), or write() is invoked.

Catalyst order: analysis (resolve columns and types) -> logical optimization -> physical planning -> whole-stage code generation; AQE may re-plan the physical plan at shuffle boundaries.

Fault tolerance: lost partitions are recomputed from lineage; a task is retried spark.task.maxFailures = 4 times per stage before the stage aborts; spark.speculation relaunches slow tasks and the first finisher wins.

Broadcast join is selected when the smaller side's estimated size is below spark.sql.autoBroadcastJoinThreshold (default 10 MB); setting it to -1 disables broadcast joins.

Defaults to memorize: spark.sql.shuffle.partitions = 200; default task parallelism = total executor cores; spark.sql.files.maxPartitionBytes = 128 MB.

SparkSession.builder().getOrCreate() is the entry point; spark.sql("SELECT ...") returns a lazily evaluated DataFrame; the same call executes DDL and registers UDFs.

CREATE TABLE without LOCATION creates a managed table whose data is deleted by DROP TABLE; adding LOCATION creates an external table whose files survive DROP TABLE.

Temporary views are session-scoped and disappear with the session; global temporary views live in the global_temp database and must be referenced as global_temp..

SparkSession.builder().enableHiveSupport() attaches to a Hive metastore so tables created outside Spark are visible; without it only the in-memory session catalog plus file-based tables exist.

spark.sql.ansi.enabled = true makes arithmetic overflow and invalid casts raise errors instead of returning null; the default in open-source Spark 3.x is false.

Hint syntax takes the form SELECT /*+ BROADCAST(t) /, /+ REPARTITION(8) /, /+ COALESCE(8) /, or /+ MERGE(t) */ and only steers the physical plan, never the results.

MERGE INTO target USING source ON key WHEN MATCHED THEN UPDATE / WHEN NOT MATCHED THEN INSERT performs upserts and needs a transactional table format such as Delta Lake.

Higher-order functions transform, filter, exists, and aggregate process array and map columns element-wise without exploding rows.

Window functions need a window spec built from partitionBy and orderBy; row_number, rank, lag, and lead reset per partition and preserve all input rows.

A DataFrame is Dataset[Row]; the typed Dataset API with case classes exists only in Scala and Java, so PySpark applications work exclusively with DataFrames.

Column references can be written col("c"), df["c"], df.c, or expr("c + 1"); lit(x) builds a literal column and selectExpr accepts raw SQL strings.

withColumn on an existing name replaces that column; withColumnRenamed renames; drop removes; withColumns (3.3+) adds several columns in a single projection.

Join types: inner, left/left_outer, right, full, left_semi, left_anti, cross; pass a list of column names for an equi-join or a Column expression for a complex condition.

repartition(n) does a full shuffle and can raise or lower partition count; coalesce(n) only merges existing partitions and cannot increase them without a shuffle.

cache()/persist() are lazy and materialize on the first action; unpersist() releases the blocks; Spark never evicts a cached DataFrame on its own.

UDF ordering by performance: built-in functions > pandas_udf (Arrow batches) > plain Python UDF (pickle per row, in a separate worker process).

Reading: spark.read.format("parquet").schema(s).load(path); inferSchema=true on CSV costs an extra full pass over the data and should be avoided in production pipelines.

explode(col) emits one row per array/map element and drops nulls; explode_outer keeps them; posexplode also returns the element position.

Skew signature: a few tasks read or write far more shuffle bytes than their peers; fixes are salting the join key, AQE skew join handling, broadcasting, or repartitioning on a better key.

Shuffle spill to disk means partitions are too few or too large; raise spark.sql.shuffle.partitions and executor memory, and lower spark.sql.files.maxPartitionBytes (default 128 MB) to read smaller splits.

Broadcast: raise spark.sql.autoBroadcastJoinThreshold above the 10 MB default only when driver and executor memory can hold the table; -1 disables broadcast joins entirely.

AQE: spark.sql.adaptive.enabled (default true since 3.2) coalesces small shuffle partitions, splits skewed ones, and can convert sort-merge joins to broadcast joins at runtime.

Driver OOM is usually collect(), toPandas(), or show() on a large result; replace with write, take(n), or a pre-aggregation before collecting.

Container killed for exceeding memory limits: increase spark.executor.memoryOverhead (default max(384 MB, 10% of executor memory)) or reduce cached data and partition size.

df.explain("formatted") reveals the physical plan; look for Exchange nodes (shuffles), SortMergeJoin versus BroadcastHashJoin, and whether filters reached the scan (predicate pushdown).

Small-file problem: many tiny files slow every read; compact with OPTIMIZE (Delta) or repartition before writing and target roughly 128 MB to 1 GB per output file.

Cache only DataFrames reused across multiple actions; caching a DataFrame read once adds memory pressure with no speedup.

A stream is an unbounded table: spark.readStream builds it and writeStream.start() launches a continuous query returning a StreamingQuery driven by awaitTermination().

Output modes: append emits only new rows (default), update emits only changed rows, complete rewrites the whole result table and is required for aggregation without a watermark.

Append mode with an aggregation requires a watermark; update mode allows streaming aggregation with or without one; complete mode discards state retention concerns.

End-to-end exactly-once needs a checkpointLocation plus a replayable source and an idempotent or transactional sink.

Watermark: df.withWatermark("eventTime", "10 minutes") lets Spark drop rows arriving later than the watermark and evicts their state, bounding memory use.

Triggers: Trigger.ProcessingTime("10 seconds") micro-batch, Trigger.AvailableNow (or once) consumes all available data then stops, Trigger.Continuous targets ~1 ms latency with at-least-once semantics.

foreachBatch(df, batchId) hands each micro-batch to a plain function, enabling MERGE/upsert and writes to sinks that are not streaming-aware.

Stateful deduplication uses dropDuplicates("id") after withWatermark; without a watermark the state store grows without limit.

Kafka source options: kafka.bootstrap.servers, subscribe or assign, startingOffsets (earliest/latest), and maxOffsetsPerTrigger to rate-limit each micro-batch.

Architecture: a thin client builds an unresolved logical plan and sends it over gRPC to a Connect server that runs the real Spark driver, optimizes the plan, and executes it on the cluster.

Client session: SparkSession.builder.remote("sc://host:15002").getOrCreate(); port 15002 is the server default and the URL scheme is sc://.

Server startup: sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_2.12:; Spark Connect requires Spark 3.4 or later.

Unsupported over Connect: SparkContext, the RDD API, spark.sparkContext, and most JVM internals; DataFrame, SQL, and UDF APIs are supported.

Sessions are isolated: each client receives its own server-side SparkSession with separate configs, temporary views, and cached DataFrames.

Client-side dependencies (Python files, archives, JARs) must be uploaded with session.addArtifact() before they can be used inside UDFs.

The pip package pyspark-client installs a lightweight client without the full Spark distribution, and client and server versions must match.

spark-submit --remote sc://host:15002 submits an application through Connect so the driver runs on the Connect server rather than on the submitting machine.

Entry point: import pyspark.pandas as ps; ps.DataFrame and ps.Series wrap a Spark DataFrame and execute lazily on the cluster, not in local memory.

Conversion: sdf.pandas_api() (also to_pandas_on_spark) returns pandas-on-Spark; psdf.to_spark() returns the Spark DataFrame; psdf.to_pandas() collects everything to the driver.

The index is not free: row-index operations such as iloc row slicing or index-based sorting trigger a shuffle; compute.default_index_type chooses distributed or distributed-sequence index generation.

compute.ops_on_diff_frames is false by default; combining two different pandas-on-Spark DataFrames requires enabling it with ps.set_option and causes a full shuffle of both frames.

compute.max_rows (default 1000) caps how many rows to_pandas and repr fetch; raise it only for small results.

Install PyArrow and set spark.sql.execution.arrow.pyspark.enabled=true to accelerate JVM-to-Python transfers used by pandas-on-Spark and pandas_udf.

Unsupported pandas methods raise PandasNotImplementedError; the recommended fallback is the equivalent Spark DataFrame operation, not a to_pandas() round trip.

pandas-on-Spark is not pandas: row order is not guaranteed, per-row Python speed is lost, and the dataset may exceed any single machine's memory.