Databricks Spark Developer Associate: Porting a Notebook to Spark Connect: Four Constructs That Cannot Cross the Wire

Porting a Notebook to Spark Connect: Four Constructs That Cannot Cross the Wire

A team moves a notebook from a co-located cluster to Spark Connect. The client is a 2-vCPU VM running PySpark 3.5 in a small container, connecting with spark = SparkSession.builder.remote('sc://gateway:15002').getOrCreate(). The notebook (a) broadcasts a 12 MB lookup dict via spark.sparkContext.broadcast(lookup), (b) calls df.rdd.map(lambda r: ...) for Python row logic, (c) reads /home/analyst/rates.csv from local disk, (d) checks a path via sc._jvm.org.apache.hadoop.fs.FileSystem, and (e) registers a UDF that imports geopandas. On the remote run it fails with AttributeError: 'SparkSession' object has no attribute 'sparkContext', an AnalysisException on the CSV path, and ModuleNotFoundError: geopandas at execution time. The purely DataFrame-based cells work fine.

1. Explain why each of the five constructs fails, and state where the client/server boundary actually sits.

Read the worked answer

The client has no Spark runtime. With Spark Connect the client builds an unresolved logical plan and ships it to the server over gRPC; execution, catalog resolution, the scheduler, the JVM and the data all live on the server.

Construct	Why it fails
`spark.sparkContext`	There is no `SparkContext` on the client. The client-side `SparkSession` is a thin facade; anything that reaches for the classic entry point (RDDs, accumulators, broadcast variables, `addFile`) is absent.
`df.rdd.map(...)`	The RDD API is not part of the Connect protocol. `df.rdd` cannot be materialised because it would require shipping partition data to the client.
`/home/analyst/rates.csv`	Paths are resolved by the server's filesystem and `fs.defaultFS`. A local path on the analyst's VM does not exist on the cluster, and the error appears at analysis time on the server.
`sc._jvm...`	There is no `sc` (no SparkContext) and no local JVM handle. Arbitrary JVM calls are not part of the protocol.
`geopandas` UDF	Python UDFs are serialised into the plan but executed by Python workers on the server. The client's installed packages are irrelevant; the library must exist on the cluster. The `ModuleNotFoundError` surfaces at execution time, which is the tell.

Boundary rule: the client may construct plans and read small results; it may not touch data, JVM objects, or the cluster filesystem.

2. Give the minimal rewrite for each construct, and flag any that you would change even if they happened to work.

Read the worked answer

(a) Broadcast variable -> broadcast join or a literal map. If the dict maps keys to values, build a small DataFrame and join with F.broadcast(small_df), or express the lookup as a CASE/create_map expression. This keeps the data on the server, where the broadcast mechanism still exists. Durability bonus: a 12 MB literal inlined into a plan is re-serialised per query - prefer the small-DataFrame join.
(b) rdd.map -> mapInPandas / applyInPandas / mapInArrow, or a plain F.udf if the logic is row-wise. Arrow-backed iterator forms avoid materialising whole partitions and are the only sane option for row-wise Python at scale.
(c) Local CSV -> upload and reference a server-visible URI. Write the file to a table, volume, or DBFS/object-store path the cluster can read, then read by URI. If the file must travel with the notebook, use the session's artifact mechanism (available in recent 3.5+ releases - verify the exact API on your build) rather than a client-local path.
(d) _jvm filesystem check -> dbutils.fs.ls-style utilities, spark.sql metadata calls, or a DataFrame input_file_name() check. Anything requiring a client-side JVM handle has no Connect equivalent by design.
(e) UDF with geopandas -> install the library on the cluster (init script / cluster library / uploaded wheel via the artifact mechanism) and pin the same minor PySpark version on client and server; version skew produces confusing serialization errors that look like UDF bugs.

Changed even if it worked: (b) - RDD-based row logic defeats Catalyst and should be Arrow-based regardless of deployment model.

3. During triage, how do you tell a client-side failure from a server-side failure, and what new operational risk does the thin client introduce?

Read the worked answer

Triage by where the artefact appears:

Client-side (plan construction): pure Python exceptions raised before any RPC completes - AttributeError, TypeError, unsupported-API errors. No Spark job, stage or SQL entry exists in the cluster UI, because no plan was submitted. Fix in the notebook.
Server-side (analysis/execution): AnalysisException (path, table, column), ModuleNotFoundError from a Python worker, task failures, spill, OOM. These have a job/stage/SQL id in the cluster UI and logs, and reach the client wrapped as a gRPC error carrying a server stack trace. Fix on the cluster (libraries, paths, resources).
Practical shortcut: reproduce with df.explain() only. If the plan builds and analysis succeeds, the remaining failures are execution-environment problems.

New operational risks:

The client is now a hard dependency for interactivity but carries no capacity. A 2-vCPU client is fine for coordination, but any .toPandas()/collect() on a wide result streams data over gRPC into that small container - the OOM moves from the cluster to the analyst's VM and becomes invisible to cluster monitoring.
Library parity is split across two machines. Client PySpark version, server PySpark version and server Python packages must be managed separately; a mismatch fails at plan-encoding time with errors that look like corrupt data.
Network partition replaces process crash. A dropped connection surfaces as a client-side transport error while the server-side query may still be running - check the cluster UI before re-submitting, or you double-run a MERGE.