Databricks Data Analyst flashcards

32 cards. Read the term and the definition here. The deck at the top of this page is optional.

Lakehouse architecture

Combines cloud object storage with Delta Lake ACID transactions and a metastore, so a single copy of data serves BI SQL, machine learning, and streaming workloads.

Medallion architecture layers

Bronze holds raw ingested data, silver holds cleaned and conformed tables, and gold holds business-level aggregates consumed by dashboards and reports.

Photon engine

A vectorized C++ query engine in the Databricks runtime and SQL warehouses that accelerates scans, joins, and aggregations over Delta tables.

Unity Catalog

A governance layer providing a catalog.schema.table namespace plus centralized privileges, lineage, auditing, and discovery across workspaces and cloud regions.

VACUUM default retention

VACUUM deletes data files no longer referenced by the Delta log; the default retention threshold is seven days, after which time travel to older versions fails.

OPTIMIZE with Z-ORDER

OPTIMIZE compacts many small files into larger ones; ZORDER BY co-locates rows with similar column values so later queries skip more files.

Managed versus external table

Managed tables keep files in the metastore's managed storage and delete them with the table; external tables only register a location and preserve the files.

Delta time travel

Query earlier table versions using VERSION AS OF or TIMESTAMP AS OF, while DESCRIBE HISTORY lists versions, operations, and per-commit metrics.

COPY INTO command

Loads files from cloud storage into a Delta table idempotently, skipping files already processed, and accepts an explicit schema or schema inference.

Auto Loader

A cloudFiles source that incrementally ingests new files from storage using directory listing or file notifications, with schema inference, schema evolution, and checkpointing.

CSV read options

Common reader options are header, delimiter, quote, escape, inferSchema, and mode, where PERMISSIVE keeps malformed rows with nulls and a corrupt record field.

External location

A Unity Catalog object pairing a storage credential with a cloud storage path so users can read and write that path without embedding secrets.

SQL warehouse sizes

Warehouse sizes run from 2X-Small to 4XLarge; each step roughly doubles compute and cost, while multi-cluster scaling adds clusters for concurrent users.

Serverless SQL warehouse

Starts in seconds, scales to zero when idle, and is fully managed by Databricks, so cost accrues only while queries actually run.

Query result cache

A warehouse returns cached results for an identical repeated query when the underlying Delta tables are unchanged, avoiding recomputation of the same work.

Parameter markers in SQL

Named placeholders written as :name inside the query editor and dashboards; key-value parameters are supplied when the query runs.

Query Profile operator tree

A visual breakdown of a query's operators showing rows, bytes, and time per node, used to locate scans, shuffles, and join bottlenecks.

Time in queue metric

How long a query waits for warehouse capacity before execution starts; high values mean the warehouse is saturated and needs scaling or a larger size.

Spilled bytes metric

Bytes written to disk because an operator exceeded available memory; any nonzero value signals a memory bottleneck to fix by repartitioning or resizing.

Dashboard dataset

A saved SQL query whose result set powers one or more visualizations on an AI/BI dashboard and is rerun when filters or parameters change.

Dashboard filter scope

A filter targets one dataset or all datasets on the dashboard, while cross-filtering narrows linked marks when a viewer clicks a data point.

Counter visualization

Displays a single aggregated number, so the underlying dataset query must already apply an aggregate function such as count, sum, or average.

Genie space instructions

Curated text and example SQL telling Genie preferred joins, filters, and business definitions, raising the accuracy of generated answers.

Genie space permissions

CAN VIEW lets users ask questions, CAN EDIT lets them curate instructions and trusted assets, and CAN MANAGE adds sharing and deletion rights.

Genie benchmarks

A set of representative questions with expected answers used to measure how often a Genie space returns correct results.

Star schema

A central fact table storing measures and foreign keys, joined to denormalized dimension tables that hold descriptive attributes for filtering and grouping.

Slowly changing dimension type 2

Preserves history by inserting a new dimension row with effective start and end dates and a current-row flag instead of overwriting the existing values.

Materialized view

Stores precomputed query results and refreshes on a schedule, incrementally where possible, giving fast answers for repeated aggregations over large tables.

Fact table grain

The meaning of one row in a fact table, such as one order line; it determines which dimensions can be joined without double counting.

Row filter in Unity Catalog

A boolean-returning function attached to a table; rows appear only when it evaluates true, enforced for every query by every user.

Column mask

A function applied to a column that returns a transformed or null value for users lacking privilege, while authorized users see the real value.

Audit log table

system.access.audit records workspace events including queries, grants, and permission changes, supporting security investigations and compliance reporting.