Databricks Intelligence Platform
Architecture
Every Databricks deployment splits into a control plane (Databricks-managed: notebook storage, job scheduler, cluster manager, Unity Catalog metadata, web UI) and a data plane (your cloud account for classic compute, or Databricks-managed for serverless). Data stays in your cloud storage; only metadata and control traffic cross into the control plane. Questions love to ask where notebooks live (control plane) versus where Spark drivers and executors run (data plane, in your VPC).
Compute choices
- All-purpose clusters: interactive, shared by people, terminated manually - expensive if left running.
- Job clusters: created for a run, terminated at completion; cheapest for scheduled work.
- SQL warehouses (classic / pro / serverless): for SQL and BI; serverless scales instantly.
- Serverless compute for notebooks and Lakeflow jobs: no cluster configuration, fastest startup, good for short spiky work.
Photon is a vectorized C++ engine that accelerates SQL and DataFrame operators on Delta tables. Trap: Photon does not speed up Python UDFs, which execute in the Python worker and often force a fallback.
Rule of thumb
Match compute to workload lifetime: if it runs on a schedule and nobody interacts with it, use a job cluster (or serverless); if an analyst clicks Run all all day, use an all-purpose cluster. Autoscaling adjusts worker count within min/max - it never scales to zero, and there is always exactly one driver.