Databricks Spark Developer Associate study plan

Diagram the driver, executors, and cluster manager flow, and mark where the Catalyst optimizer builds and optimizes a query plan

Trace one DataFrame action end to end into jobs, stages, and tasks, and identify which transformations force a shuffle

Contrast narrow and wide transformations with at least three concrete examples of each

Explain lazy evaluation by showing what runs when a transformation is called versus when an action is called

Identify which Spark UI tabs reveal stage boundaries, task skew, and shuffle volume

Read CSV, Parquet, and JSON into DataFrames using explicit schemas rather than schema inference

Apply select, filter, withColumn, drop, distinct, and orderBy to reshape a dataset

Execute inner, left, and left-semi joins and verify row counts against manually computed expectations

Handle missing data with na.drop, na.fill, and coalesce, and state the row-count effect of each

Convert five imperative row-by-row loops into column expressions

Register DataFrames as temporary views and query them with Spark SQL

Write queries using joins, GROUP BY, HAVING, subqueries, and CTEs

Apply built-in date, string, and conditional (CASE/WHEN) functions to transform columns

Read EXPLAIN output and label the scan, join, aggregate, and exchange nodes in a physical plan

Rewrite five DataFrame operations as equivalent SQL and compare the resulting plans

Implement window functions (row_number, rank, lag, lead, running totals) in both the DataFrame API and Spark SQL

Flatten and transform nested columns using explode, arrays_zip, and higher-order functions such as transform and filter

Distinguish repartition from coalesce and predict the resulting partition count for each call

Compare a Python UDF, a pandas UDF, and a built-in function on the same task for correctness and speed

Explain why typed Datasets exist only on the JVM and PySpark exposes the untyped DataFrame API

Diagnose shuffle spill and partition skew from Spark UI task duration and shuffle read size metrics

Apply a broadcast hint, key salting, and adaptive query execution settings to repair a skewed join

Tune read and write partition counts by sizing partitions against target file sizes

Decide when to cache or persist a DataFrame, select a storage level, and identify cases where caching costs more than it saves

Measure elapsed time before and after each tuning change on a synthetic skewed dataset

Read from a file source and a rate source, and write to a file sink with a checkpoint location configured

Select a trigger mode and an output mode for a given use case and justify both choices

Apply a watermark to bound state and confirm that late-arriving rows are dropped rather than aggregated

Implement a stateful aggregation and explain how checkpointing plus an idempotent sink yields exactly-once semantics

Inspect StreamingQueryProgress metrics to report input rate, processed rows, and watermark lag

Open a remote Spark session through Spark Connect or Databricks Connect and run a DataFrame job on it

List Spark Connect constraints, including client-server separation, unsupported APIs, and version compatibility requirements

Convert three pandas scripts to the pandas API on Spark using pyspark.pandas

Run groupby and apply through pandas-on-Spark and identify when native PySpark outperforms it

Explain how the default index and single-machine pandas semantics differ from distributed execution

Complete a timed 60-question mock exam spanning all seven domains

Re-drill every domain scored below 80 percent until the missed concept is explained without notes

Write out five tuning scenarios, naming the symptom, the metric that reveals it, and the fix

Distill the highest-yield facts into a one-page reference sheet covering partitioning, shuffles, watermarks, and Connect