Databricks Spark Developer Associate key terms

The JVM process that hosts the SparkSession, converts your code into a query plan, schedules work on executors, and returns results to the user. If the driver stops, the whole application stops.

A JVM process on a worker node that runs the individual tasks of an application and stores cached partitions. Each executor has a fixed number of cores and an amount of memory.

The entry point object for Spark programming, giving access to DataFrame, SQL and catalog APIs plus runtime configuration. In Databricks notebooks it already exists as the variable spark.

A job is the work triggered by one action; it is split into stages at shuffle boundaries; each stage runs one task per partition. Tasks are the smallest unit of scheduled work.

A chunk of a distributed collection that one task processes. The number of partitions sets the maximum parallelism, and their size balance determines whether tasks run evenly.

The redistribution of rows across executors by key or by partition, required by wide operations such as groupBy, join, and distinct. Shuffles write to disk and move data over the network.

Narrow transformations such as select or filter compute each output partition from a single input partition. Wide transformations such as groupBy or join read many input partitions and therefore force a shuffle.

The graph of stages Spark derives from a query plan. The DAG scheduler submits stages in dependency order, retries failed tasks, and recomputes partitions whose data was lost.

Transformations only record a plan; nothing executes until an action such as count, show, or write is called. This lets the optimizer improve the entire chain before running it.

Spark SQL's query optimizer. It applies rule-based rewrites and cost-based decisions to logical plans, then selects a physical plan, so declarative queries often beat hand-written procedural code.

Spark's physical execution layer, which uses whole-stage code generation, off-heap binary data formats, and cache-aware algorithms to reduce CPU and memory overhead during task execution.

A distributed table of rows with named, typed columns and a logical plan. It is immutable; each transformation returns a new DataFrame that the optimizer can rewrite before execution.

A typed API available in Scala and Java where rows map to case classes or beans, giving compile-time checking while still using the Catalyst optimizer and Tungsten engine.

The schema class that describes a DataFrame as a list of fields, each with a name, data type, and nullability flag. It is used to declare schemas explicitly when reading data.

Spark's automatic guessing of column names and types by scanning sample data. It costs an extra read and can pick wrong types, so production pipelines usually declare schemas.

A function written in Python and registered for use inside Spark SQL or DataFrame expressions. Rows are serialized one at a time between the JVM and Python, so it is slow.

A user-defined function that receives data as Arrow-backed pandas Series or DataFrames in batches instead of one row at a time, greatly reducing serialization overhead compared with a plain Python UDF.

A join that copies the small input to every executor so the large input needs no shuffle. Spark chooses it from table statistics, a broadcast hint, or adaptive query execution.

Storing a DataFrame's computed partitions on executors, in memory, disk, or both, so repeated actions skip recomputation. Useful when the same result is reused by several steps.

A runtime feature that re-optimizes the plan between shuffle stages using real data sizes: it merges small partitions, switches to broadcast joins, and splits skewed partitions.

A condition where a few partition keys hold far more rows than others, so a handful of tasks run long while most executors idle. Common fixes are salting, broadcast joins, and AQE skew handling.

The writing of intermediate data to executor local disk when memory is full. It prevents out-of-memory failures but adds I/O and slows tasks noticeably.

The web interface for a running or finished application, showing jobs, stages, tasks, SQL plans, shuffle reads and writes, and storage. It is the first place to diagnose slow stages.

Optimizations that move filters down to the data source so only matching partitions, files, or row groups are read. They are the main reason partitioned and Delta tables scan quickly.

Databricks' centralized governance layer, naming data as catalog.schema.table and adding permissions, auditing, and lineage across workspaces. Both SQL and DataFrame code resolve names through it.

For a managed table, the metastore owns both metadata and files, so dropping it deletes the data. For an external table, files live at a path you choose and survive the drop.

A named SQL view over a DataFrame that lives only for the current session. Global temporary views are visible across sessions of one application under the global_temp database.

A Delta Lake statement that inserts, updates, and deletes target rows in a single atomic operation based on a match condition with a source table. It is the standard way to upsert.

Delta Lake commands: OPTIMIZE compacts many small files into larger ones, and Z-ORDER clusters rows by chosen columns so later filtered queries skip more files.

An unbounded table evaluated incrementally, where each trigger processes only newly arrived data. Results are written with writeStream and require a checkpoint location for recovery.

Triggers decide when micro-batches run, for example every few seconds, once, or availableNow. Output modes decide what each batch emits: append only, update changed rows, or complete results.

A bound on how late event-time data may arrive. Spark uses it to drop old rows and to limit the state kept for streaming aggregations, enabling append output.

A durable directory where a streaming query records offsets, commit logs, and aggregation state. Restarting from the same checkpoint resumes exactly where the query stopped.

A Databricks source, used with format cloudFiles, that discovers new files in cloud storage incrementally and tracks them, with schema inference and evolution, without listing the whole directory.

A streaming sink option that hands each micro-batch to your function as a static DataFrame, so you can run MERGE, write to several sinks, or reuse batch code.

A client-server architecture in which a thin client sends unresolved plans over gRPC to a separate Spark server. This decouples the user's environment and language from the cluster.

A SparkSession created by a Spark Connect client that points at a server URL instead of starting a local JVM. API calls become serialized plans executed on the server.

A pandas-compatible interface (pandas-on-Spark, formerly Koalas) that runs familiar pandas code on distributed Spark data using ps.DataFrame and ps.Series, imported from pyspark.pandas.

In pandas API on Spark, data is split across partitions, each with its own index, so index-based operations such as iloc or some joins differ from single-machine pandas and may require a shuffle.

Methods for moving between APIs: to_spark() and to_pandas_on_spark() switch representations, while to_pandas() collects a distributed frame into local memory and should only be used on small results.