Understanding the Databricks Data + AI Platform
The lakehouse in one paragraph
Databricks combines a data lakehouse - cheap cloud object storage (S3, ADLS, GCS) managed in Delta Lake format - with a managed control plane that runs workspaces, notebooks, jobs, and metadata. The critical split: the control plane lives in Databricks' account; the data plane (your data and often the compute VMs) lives in your cloud account or, for serverless, in Databricks-managed infrastructure inside your region.
Core building blocks you must be able to name
- Delta Lake - ACID transactions, schema enforcement, time travel on top of Parquet files.
- Unity Catalog - the governance layer: three-level namespace
catalog.schema.table, grants, lineage, audit, row filters and column masks. - Compute types - all-purpose clusters (interactive, shared), job clusters (ephemeral, cheaper), and SQL warehouses (serverless, pro, classic) tuned for BI and Databricks SQL.
- Workload surfaces - notebooks and Workflows/Lakeflow for pipelines; Databricks SQL editor, dashboards, and AI/BI Genie for analytics.
Traps candidates hit
- Assuming SQL warehouses can run Python notebooks - they cannot; they run SQL only.
- Confusing workspace admin with account admin or metastore admin.
- Believing data is stored in Databricks' account - it is in object storage you control.
- Forgetting that Unity Catalog privileges are inherited downward from catalog to schema to table.
Rule of thumb for choosing compute
Ask who waits? If a human waits interactively, use an all-purpose cluster or a SQL warehouse. If nobody waits, use a job cluster scheduled by Workflows. For a dashboard viewed by 50 people, a small serverless SQL warehouse with auto-scaling beats a big classic warehouse - concurrency, not data volume, drives sizing.
Worked example
A team ingests CSV files nightly, refines them, and publishes a dashboard. Bronze/silver pipelines → Workflows on job clusters. Gold tables queried by the dashboard → serverless SQL warehouse. Governance across both → Unity Catalog.