Databricks Spark Developer Associate flashcards

31 cards. Read the term and the definition here. The deck at the top of this page is optional.

Driver program

Process that creates the SparkSession, builds the DAG of transformations, schedules jobs with the cluster manager, and coordinates executors; it must stay alive for the whole application.

Executor

JVM process launched on a worker node that runs tasks in threads and can cache data; each executor has a fixed number of cores acting as task slots plus allocated memory.

Narrow dependency

Relationship where each parent partition feeds at most one child partition, as in map or filter; no data crosses the network and no new stage is created.

Wide dependency

Relationship where one child partition depends on many parent partitions, as in groupBy or a non-broadcast join; Spark shuffles data across executors and starts a new stage.

Job, stage, task

An action creates a job; the job is divided into stages at shuffle boundaries; each stage runs one task per partition of its input.

Managed table

Table whose metadata and data files are both owned by the metastore catalog; running DROP TABLE deletes the underlying data as well as the catalog entry.

External table

Table registered with an explicit LOCATION; DROP TABLE removes only the metastore entry and leaves the data files intact on the underlying storage system.

Global temporary view

View stored in the global_temp database and referenced as global_temp.name; it is shared across sessions of one application and dropped when the application ends.

Query hints

Comments such as /*+ BROADCAST(t) */ or /*+ REPARTITION(8) */ placed after SELECT that steer the optimizer's physical plan without changing the query's results.

MERGE INTO

SQL statement that upserts source rows into a target using WHEN MATCHED THEN UPDATE and WHEN NOT MATCHED THEN INSERT; it requires a transactional table format such as Delta Lake.

withColumn on existing name

Calling withColumn with a column name that already exists replaces that column instead of adding a duplicate; use withColumnRenamed to rename and drop to remove.

repartition versus coalesce

repartition(n) shuffles all data and can increase or decrease partitions; coalesce(n) only merges existing partitions and cannot increase the count without a shuffle.

Python UDF cost

A plain Python UDF serializes every row with pickle to a separate Python worker process, so it is far slower than built-in functions and blocks Catalyst code generation.

pandas_udf

Vectorized user-defined function that uses Apache Arrow to transfer batches of data between the JVM and Python, delivering much higher throughput than a row-at-a-time UDF.

Data skew

Condition where a few shuffle partitions hold far more rows than the rest, so a handful of tasks dominate stage runtime; remedies are salting, AQE skew handling, or broadcasting.

Shuffle spill

When sort or shuffle buffers exceed available memory, Spark writes the overflow to disk; heavy spill indicates too few or oversized partitions and sharply slows the stage.

Adaptive Query Execution

Runtime re-optimization at shuffle boundaries that coalesces small partitions, splits skewed partitions, and can convert sort-merge joins to broadcast joins; controlled by spark.sql.adaptive.enabled.

Broadcast join threshold

spark.sql.autoBroadcastJoinThreshold, default 10 MB; a table estimated below this size is broadcast to every executor instead of being shuffled, and -1 disables the feature.

Driver out-of-memory

Usually caused by collect(), toPandas(), or show() pulling an entire result set into the driver process; use write, take(n), or aggregated output instead.

Output modes

Append writes only new rows, update writes only changed rows, and complete rewrites the entire result table; complete is required for aggregation with no watermark.

Watermark

withWatermark(column, delay) declares how late data may arrive; rows older than the watermark are dropped and their state evicted, which bounds streaming memory usage.

Checkpoint location

Directory named by checkpointLocation that stores offsets, state, and query metadata; it is required for stateful operations and for exactly-once recovery after a failure.

foreachBatch sink

Method that gives each micro-batch to a user function as a DataFrame plus a batch id, enabling MERGE or upsert logic and writes to non-streaming sinks.

Spark Connect architecture

A thin client sends an unresolved logical plan over gRPC to a server running the real Spark driver, which optimizes, plans, and executes it on the cluster.

Remote session URL

SparkSession.builder.remote("sc://host:15002").getOrCreate() creates a client session bound to a Spark Connect server listening on the default port 15002.

Connect API limitations

Spark Connect supports DataFrame, SQL, and UDF APIs but not SparkContext, the RDD API, or most JVM internals, so RDD-based code must be rewritten first.

addArtifact

Session method that uploads client-side Python files, archives, or JARs to the Spark Connect server so UDFs and their dependencies resolve there.

pandas-on-Spark import

import pyspark.pandas as ps provides ps.DataFrame and ps.Series, distributed equivalents of pandas objects backed by a Spark DataFrame and evaluated lazily.

to_spark and pandas_api

psdf.to_spark() returns the underlying Spark DataFrame, sdf.pandas_api() wraps a Spark DataFrame as pandas-on-Spark, and to_pandas() collects results to the driver.

Default index type

Spark has no natural row order, so pandas-on-Spark generates a synthetic index; compute.default_index_type selects distributed or distributed-sequence, and index operations trigger shuffles.

compute.ops_on_diff_frames

Option that is false by default; setting it true allows combining two different pandas-on-Spark DataFrames, an operation that forces a full shuffle of both frames.