Databricks Spark Developer Associate: Developing DataFrame/Dataset API Applications

Core model

A DataFrame is Dataset[Row] - untyped, optimised by Catalyst. A Dataset[T] with a case class or bean is typed and available only in Scala/Java; Python and R only have DataFrames. Build one with spark.read.*, spark.createDataFrame(rows, schema), or spark.range(n). Prefer supplying an explicit StructType: schema inference over a Python list of tuples triggers an extra job over the data.

Transformations you must know

Column selection and derivation (select, selectExpr, withColumn, withColumnRenamed, drop), filtering (filter/where, isin, between, like), null handling (na.drop, na.fill, na.replace, coalesce, when/otherwise), aggregation (groupBy().agg(), pivot, agg with dicts), deduplication (dropDuplicates, distinct), and joins (inner, left, right, full, left_semi, left_anti, cross).

Traps

• left_semi / left_anti return only left-side columns - a very common wrong answer.
• Chaining withColumn inside a loop grows the logical plan linearly; a single select with many expressions is far cheaper.
• Python UDFs are black boxes to Catalyst: row-by-row (de)serialisation, no predicate pushdown. Use built-in pyspark.sql.functions, or a pandas UDF with Arrow for vectorised work.
• After a join, df1.id and df2.id collide; alias the DataFrames or use df1["id"].
• select("a") on an ambiguous name raises AnalysisException; withColumn with an existing name silently replaces it.

Rule of thumb

Prefer F.col("name") over attribute access, and prefer chained built-ins over UDFs. Example: df.withColumn("flag", F.when(F.col("amt") > 100, 1).otherwise(0)) stays fully in the JVM and is pushed down where possible.