Databricks Spark Developer Associate common mistakes
Databricks Spark Developer Associate · Understand the traps and how to avoid them.
Assuming a DataFrame transformation filters or reduces data immediately - e.g. writing `df.filter(...)` and then complaining that the Spark UI shows the same input bytes read for the raw DataFrame.
Candidates carry over eager-evaluation intuition from pandas/SQL clients and miss that DataFrames are lazy; only actions (show, count, write, collect) trigger a job, and each action re-runs the full lineage unless cached.
What to do instead
Classify every operation as transformation vs. action, trace the plan with df.explain() before and after, and call .cache()/.persist() on any DataFrame reused across multiple actions.
Confusing `repartition(n)` with `coalesce(n)` - using `coalesce` to increase parallelism, or `repartition` casually on a huge dataset to 'fix' skew.
Both take a partition count and look interchangeable in autocomplete; the shuffle/no-shuffle distinction and the resulting partition-size imbalance are easy to gloss over when studying.
What to do instead
Memorise: repartition does a full shuffle and may go up or down; coalesce only merges existing partitions and cannot increase parallelism. Use coalesce to shrink before writing; use repartition(col) or salting to actually redistribute skewed keys.
Writing UDFs (especially Python UDFs) for logic that native Spark functions already support, then failing questions about why the job is slow or why a Catalyst optimization no longer applies.
UDFs are the most familiar tool for people coming from pandas, and the exam's performance questions are usually testing whether you know UDFs are opaque to Catalyst and force serialization between the JVM and Python worker.
What to do instead
Default to pyspark.sql.functions for anything non-trivial; if a UDF is unavoidable, use a pandas (vectorized) UDF or register a Scala/Java UDF, and be able to state that regular Python UDFs block Catalyst codegen and add per-row serde overhead.
Treating a streaming DataFrame like a batch DataFrame - calling `count()` or `collect()` on it, or forgetting that `df.write` needs `.format("delta").start(path)` rather than `.save(path)`.
Structured Streaming reuses the batch DataFrame API, so syntax looks nearly identical; the trigger/checkpoint/output-mode machinery only matters at the action, so candidates copy batch snippets verbatim.
What to do instead
For streaming sources, always end with writeStream plus outputMode, option("checkpointLocation", ...) and a trigger, then .start(). Never call batch actions on a streaming DataFrame - use display/writeStream instead.
Forgetting the checkpoint location on a streaming query, or reusing one checkpoint directory across two different queries.
Checkpoints are invisible in quickstart examples that use the default, and candidates assume they are optional logs rather than the source of truth for offsets and state.
What to do instead
Treat checkpointLocation as mandatory and unique per query (including per sink); store it in durable cloud storage, and remember changing the query logic generally requires a fresh checkpoint directory.
Mixing up `complete`, `append` and `update` output modes, e.g. choosing `append` with an aggregation and expecting to see rows change, or `complete` with a non-aggregating query.
Output modes only bite in combination with aggregation and watermarking rules, and candidates memorise the three names without the compatibility matrix behind them.
What to do instead
Anchor on behaviour: append adds only new finalised rows (requires watermark for aggregations), update emits only changed rows, complete rewrites the whole result table (aggregations only, no watermark-dropped rows). Check the mode against the query shape before answering.
Answering driver/memory questions by adding executors or raising `spark.executor.memory`, when the actual failure is driver-side (e.g. `collect()` of a large result, or a broadcast join of a huge table).
Learners learn 'Spark is distributed' and reach for cluster size; distinguishing driver heap, executor heap, and broadcast limits requires knowing where each operation runs.
What to do instead
Localise first: wide collect/toPandas and broadcast variables live on the driver (spark.driver.memory, spark.driver.maxResultSize); task OOM and spilled shuffle live on executors. For joins, compare table sizes against spark.sql.autoBroadcastJoinThreshold before touching memory settings.
Assuming the number of output files equals the number of partitions after a `groupBy` - or being surprised that a small result is written as 200 tiny files.
The shuffle partition count default and its interaction with coalesce/repartition before writing is a purely architectural fact that is easy to read past.
What to do instead
Remember a shuffle produces spark.sql.shuffle.partitions partitions (default 200) regardless of key count; call coalesce(n) or repartition(n) on the result before write when file count matters, and check df.rdd.getNumPartitions() empirically.
Trying to use `spark.sparkContext` (RDD APIs, `parallelize`, `broadcast` off the context) or `sparkContext.setLogLevel` idioms inside a Spark Connect session and getting attribute errors.
Almost all tutorials predate Spark Connect; the classic SparkContext is not available on the client side because the client only speaks the gRPC/DataFrame protocol to a remote server.
What to do instead
In Connect sessions, stay in the DataFrame/spark.sql API surface and use spark.conf for configuration. Know that a small allowlist of client-side artifacts (e.g. UDFs registered via spark.udf.register and spark.addArtifacts) is shared, but JVM-level SparkContext calls fail by design.
Using pandas-on-Spark like local pandas - calling `to_pandas()` on a big distributed frame, or doing per-row `apply` loops and expecting pandas speed.
The pyspark.pandas import is a drop-in alias, which hides the fact that operations translate to Spark jobs; some ops (e.g. certain apply, iterrows, non-index-aligned joins) trigger costly shuffles or full materialisation.
What to do instead
Keep data in the pandas API on Spark and only convert at the boundaries; prefer built-in vectorised pandas-API methods over apply, set a distributed index before joins, and remember to_pandas() pulls everything to the driver - use to_spark() to return to a DataFrame instead.