I85PREP
⌘ K
Databricks Spark Developer Associate study materials
Databricks Spark Developer Associate study resources

Databricks Spark Developer Associate: Using Spark Connect to Deploy Applications

Databricks Spark Developer Associate · Work through your exam, one topic at a time.

All guide sections

Using Spark Connect to Deploy Applications

What it is

Spark Connect decouples the client from the cluster. The client runs a thin library with no JVM and no Spark driver; it builds the same DataFrame logical plan, serialises it to Protobuf, and ships it over gRPC to a Spark server that executes it. Databricks Connect is built on this architecture, letting you run pyspark-style code in an IDE or notebook against a remote cluster.

Connecting

from pyspark.sql import SparkSession
spark = SparkSession.builder.remote("sc://my-host:443/;token=...").getOrCreate()

Or set spark.remote in configuration. spark.conf, spark.sql, spark.catalog, spark.read/write, and the full DataFrame API work as usual.

Capabilities and limits

The API is DataFrame-first. Supported: SQL, DataFrame transformations, UDFs (serialised and shipped), foreachBatch, Pandas API on Spark. Limited or unsupported: the RDD API, direct SparkContext access (e.g. sc.parallelize, sc.setLogLevel, accumulator tricks), some spark.sparkContext calls, and libraries that expect a local JVM driver. Client-side Python dependencies must be shipped to the server with spark.addArtifact(...) before you import them in a UDF.

Traps

  • Calling .rdd or spark._sc on a Connect session raises an error - questions often hide this inside otherwise valid code.
  • Results still come back over the network: collect()/toPandas() on a large DataFrame overloads the client.
  • Spark UI access is not local; you view it on the cluster/workspace.
  • Long-running interactive sessions keep the remote cluster alive - stop them.

Rule of thumb

Write your code exactly as you would locally, then audit for anything touching SparkContext, RDD, or local files that the server cannot see.