Databricks Spark Developer Associate case studies
Databricks Spark Developer Associate · Put your knowledge to work in context.
The Databricks Certified Associate Developer for Apache Spark is a proctored, multiple-choice certification that assesses understanding of Apache Spark architecture and components plus the ability to apply the Spark DataFrame API to complete basic data manipulation tasks within a Spark session. The published scope includes selecting, renaming and manipulating columns; filtering, dropping, sorting and aggregating rows; handling missing data; combining, reading, writing and partitioning DataFrames with schemas; and working with UDFs and Spark SQL functions. The Databricks Spark Developer Associate case studies are 5 cases and about 4,495 words.
1The 4,000-Task Stage That Nobody Can Speed Up With More Executors
A nightly job reads a 4,000-file Parquet directory (~1.8 TB), applies a handful of narrow withColumn transformations, and writes 4,000 output files. No shuffle occurs anywhere in the job.
857 words, about 4 min2Twelve Tasks That Take 55 Minutes: Skew, AQE, and a Tempting Broadcast
A daily query joins orders (2.1 TB, 40,000 files) to customers (6 GB, 41 million rows) on customer id . The join shuffles into 192 partitions.
877 words, about 4 min3A Streaming Job That Never Catches Up - and Silently Drops Data
A Structured Streaming job reads a 12-partition Kafka topic peaking at 9,000 messages/s and writes to Delta through foreachBatch , running a MERGE on message id. Configuration: trigger(processingTime='10 seconds') , no maxOffsetsPerTrigger , spark.sql.shuffle.partitions=200 , and withWatermark('event time', '30 minutes') followed by dropDuplicates('event id') that the team believes performs one-hour deduplication.
918 words, about 5 min4Porting a Notebook to Spark Connect: Four Constructs That Cannot Cross the Wire
A team moves a notebook from a co-located cluster to Spark Connect. The client is a 2-vCPU VM running PySpark 3.5 in a small container, connecting with spark = SparkSession.builder.remote('sc://gateway:15002').getOrCreate() .
911 words, about 5 min5Pandas API on Spark: Six Groups, One Model, and a Very Long Night
A data scientist must enrich a 900 GB Parquet table using a scikit-learn model (a 400 MB joblib file) loaded at module import. The code is psdf.groupby('region').applyInPandas(enrich, schema=out schema) over 6 regions; the largest region holds 62% of the rows.
932 words, about 5 min