I85PREP
⌘ K
Databricks Data Analyst study materials
Databricks Data Analyst study resources

Databricks Data Analyst: Managing Data

Databricks Data Analyst · Work through your exam, one topic at a time.

All guide sections

Managing Data

Managed versus external tables

The single most tested distinction. A managed table's data files are owned by Unity Catalog and deleted when you DROP TABLE. An external table points at a storage location you specify; DROP TABLE removes only the metadata, leaving files intact. If data must outlive the table, make it external.

Delta Lake operations to know

  • DESCRIBE HISTORY / time travel with VERSION AS OF or TIMESTAMP AS OF.
  • OPTIMIZE compacts small files; OPTIMIZE ... ZORDER BY (col) co-locates related data; liquid clustering is the modern alternative and does not require partitioning.
  • VACUUM deletes unreferenced files older than the retention threshold (default 7 days). Once vacuumed, time travel to older versions fails - a favorite trick question.
  • Constraints: NOT NULL, CHECK, and informational PRIMARY KEY/FOREIGN KEY (not enforced for uniqueness).

Architecture patterns

The medallion layout - bronze (raw), silver (cleaned/conformed), gold (business-level aggregates) - is the expected answer whenever a question asks how to organize a lakehouse. Each hop should be a Delta table so downstream consumers get schema enforcement and lineage.

Traps

  1. Dropping an external table and expecting files to disappear - they do not.
  2. Running VACUUM then trying to query an old version.
  3. Assuming a primary key prevents duplicate inserts; enforce with MERGE or DROP DUPLICATES.
  4. Over-partitioning small tables, producing thousands of tiny files.

Rule of thumb

Cluster or partition on columns used in high-selectivity filters, and only when the table holds roughly a terabyte or more. Below that, liquid clustering plus OPTIMIZE is simpler and usually faster.

Worked example

A landing zone receives daily files. Create an external Delta table over abfss://raw/sales/, then build a silver managed table with MERGE INTO keyed on order_id. Schedule OPTIMIZE weekly; never VACUUM with zero retention on a table you may need to restore.