Databricks Spark Developer Associate study guide
Databricks Spark Developer Associate · Work through your exam, one topic at a time.
The Databricks Certified Associate Developer for Apache Spark is a proctored, multiple-choice certification that assesses understanding of Apache Spark architecture and components plus the ability to apply the Spark DataFrame API to complete basic data manipulation tasks within a Spark session. The published scope includes selecting, renaming and manipulating columns; filtering, dropping, sorting and aggregating rows; handling missing data; combining, reading, writing and partitioning DataFrames with schemas; and working with UDFs and Spark SQL functions. The Databricks Spark Developer Associate study guide has 7 sections and about 2,235 words.
01Spark Architecture and Components
An application is a driver (runs main() , owns the SparkSession / SparkContext , builds the DAG, schedules work, collects results) plus a set of executors (JVM processes that run tasks in core slots and hold cached blocks). A cluster manager - Standalone, YARN, Kubernetes, or Databricks' managed layer - allocates executors.
315 words, about 2 min02Using Spark SQL
spark.sql("...") returns a DataFrame. You work with databases, managed vs external tables, and views: Temp views are scoped to the SparkSession that created them.
313 words, about 2 min03Developing DataFrame/Dataset API Applications
A DataFrame is Dataset[Row] - untyped, optimised by Catalyst. A Dataset[T] with a case class or bean is typed and available only in Scala/Java; Python and R only have DataFrames.
329 words, about 2 min04Troubleshooting and Tuning DataFrame Applications
Read .explain(True) for the parsed, analysed, optimised and physical plans. Look for Exchange (shuffle), BroadcastHashJoin vs SortMergeJoin , FileScan with pushed filters, and AdaptiveSparkPlan .
325 words, about 2 min05Structured Streaming
spark.readStream → DataFrame (an unbounded table) → transformations → writeStream . Common sources are Kafka, Delta Lake, and file/JSON/CSV directories; sinks are Delta/file, Kafka, foreachBatch , and memory/console for debugging.
297 words, about 1 min06Using Spark Connect to Deploy Applications
Spark Connect decouples the client from the cluster. The client runs a thin library with no JVM and no Spark driver ; it builds the same DataFrame logical plan, serialises it to Protobuf, and ships it over gRPC to a Spark server that executes it.
327 words, about 2 min07Using the Pandas API on Spark
The Pandas API on Spark ( pyspark.pandas , conventionally imported as ps ) lets you run most of the pandas API on a distributed Spark DataFrame. ps.DataFrame and ps.Series are partitioned; ps.read csv , ps.read parquet , and ps.sql("SELECT ...") are the entry points.
329 words, about 2 min