Developing Code for Data Processing using Python and SQL
Choosing the right tool inside the engine
Most marks here are lost by reaching for a user-defined function when a built-in exists. Before writing a UDF, check pyspark.sql.functions (including higher-order functions such as transform, filter, exists, aggregate, and try_* variants for safe casting). Built-ins run inside the JVM; ordinary Python UDFs force row-by-row serialisation between the JVM and a Python worker, which is the classic exam trap when a question asks why a pipeline slowed down after refactoring.
When Python is unavoidable, prefer Arrow-optimised Python UDFs or vectorised pandas UDFs (applyInPandas, mapInPandas), which batch rows through Apache Arrow. Remember that a pandas UDF's schema must be declared explicitly for applyInPandas, and that grouped applyInPandas moving a whole group to one worker is a memory hazard.
Laziness, control flow, and parameters
Transformations are lazy; try/except around a select catches nothing until an action (or cache) triggers execution - wrap the action. In SQL, distinguish temporary views (session-scoped) from global temporary views and CTEs, and know that Databricks SQL supports compound statements (BEGIN … END, DECLARE, IF, FOR) for scripting, plus named parameter markers (:param) and widgets/dbutils.widgets for parameterising notebooks and jobs.
Rule of thumb: if a column expression can be written with built-in functions, never ship a Python UDF - and when you must, hand it Arrow.
Test small DataFrames locally with pytest against a Spark session rather than debugging in a cluster loop.