Using the Pandas API on Spark
What it is
The Pandas API on Spark (pyspark.pandas, conventionally imported as ps) lets you run most of the pandas API on a distributed Spark DataFrame. ps.DataFrame and ps.Series are partitioned; ps.read_csv, ps.read_parquet, and ps.sql("SELECT ...") are the entry points.
Converting
psdf.to_spark()(orspark.createDataFrame(psdf)) - stays distributed.psdf.to_pandas()- collects everything to the driver; the usual cause of client OOM.spark_df.pandas_api()- goes the other way.
Options you should know
ps.set_option("compute.default_index_type", ...) controls the default index. "distributed" is cheap but the index repeats across partitions; "distributed-sequence" yields a true global sequential index at the cost of a shuffle (sequence in some versions is a middle ground). ps.options.compute.max_rows limits how many rows .head()/printing displays. .plot is available via the configured plotting backend.
Traps
- The API is not complete pandas. Unsupported methods raise
PandasNotImplementedError; a question offering a pandas one-liner that has no distributed equivalent is usually the wrong answer. - Operations that require a whole column in memory (some
apply, sorting on a huge frame,.iterrows()) either fail or warn - vectorise instead of row-looping. - The index is not a pandas index:
reset_index()/set_index()behave differently and joins on index can trigger unexpected shuffles. psDataFrames are lazy, like Spark:head()returns results but many methods are transformations.
Rule of thumb
Use ps to migrate existing pandas code with minimal edits, call .to_spark() before anything performance-critical, and only call .to_pandas() after filtering or aggregating down to a size that fits the driver.