IBM watsonx Data Lakehouse Engineer Associate flashcards

32 cards. Read the term and the definition here. The deck at the top of this page is optional.

watsonx.data lakehouse

A data architecture combining data lake storage (object storage) with data warehouse query engines, using open table formats and a shared metastore to avoid data duplication.

Presto coordinator

The Presto node that parses SQL, plans queries, and schedules tasks across workers. It does not execute data processing itself in watsonx.data deployments.

Catalog in watsonx.data

A namespace binding a storage location and a metastore. Multiple catalogs let one Presto engine query data across different buckets and sources.

Hive metastore

A service storing table definitions and partition metadata. watsonx.data can use an existing Hive metastore or provision its own to share tables across engines.

Db2 Warehouse engine

An optional MPP relational engine in watsonx.data for high-concurrency BI and SQL workloads. It can query object storage through external tables.

Infrastructure Manager

The watsonx.data console page for viewing and managing engines and storage. It shows engine health, pod status, and allows adding or scaling components.

Schema equals database

In watsonx.data, a database object corresponds to a schema. Tables are created inside a schema within a catalog, and SQL references catalog.schema.table.

BYOC deployment

Bring-your-own-cloud: watsonx.data deployed into a customer's AWS account via the marketplace, giving control over networking and storage while IBM manages the software.

Register object storage

The act of adding a bucket (COS, S3, Azure Blob) to watsonx.data by supplying endpoint, credentials, and a catalog name so engines can read its data.

Presto JDBC connector

A Presto catalog configuration that connects to an external relational database (PostgreSQL, MySQL, Db2, Oracle) so its tables can be queried without copying data.

Partitioned external table

A table whose data is stored in Hive-style directories (col=value). Declaring partition columns in CREATE TABLE enables partition pruning during queries.

Ingestion job API

A REST interface that defines source, target catalog/schema/table, file format, and write mode (append/overwrite) to load data into watsonx.data programmatically.

Delta Lake support

watsonx.data can read and write Delta Lake tables, providing ACID transactions and time travel on object storage alongside Parquet and ORC formats.

HMAC keys for COS

Access key and secret key credentials used to authenticate to IBM Cloud Object Storage when registering a bucket as a watsonx.data storage source.

Schema evolution refresh

After adding columns to external Parquet/ORC files, the metastore must be refreshed (e.g., ALTER TABLE REFRESH) so queries see the new columns.

Federated query

A single SQL statement that joins data across multiple connectors (e.g., object storage and a PostgreSQL database) without moving or copying the data first.

Write modes append/overwrite

Ingestion job options: append adds rows to an existing table; overwrite replaces the table contents. The mode is specified in the job definition.

Presto broadcast join

A join strategy where a small table is replicated to all workers. Set join distribution to BROADCAST to avoid shuffling large tables across the network.

EXPLAIN plan

A Presto statement that displays the logical and distributed query plan. Use it to check join order, filters pushed down, and stage structure before tuning.

Predicate pushdown

An optimization where filters in a query are pushed to the data source (e.g., JDBC connector or Parquet reader) so fewer rows are read and transferred.

ANALYZE table

A command that collects table and column statistics in the metastore. Presto's cost-based optimizer uses these statistics to choose join order and strategies.

Small file problem

Many tiny files increase task overhead and metadata load. Target 128–512 MB per Parquet file and compact small files with CTAS or Spark jobs.

Presto resource groups

Configuration that limits concurrent queries and memory per workload group, isolating BI dashboards from heavy ETL queries on the same engine.

Spark engine use cases

In watsonx.data, Spark handles ETL, machine learning, and complex transformations. It reads and writes the same catalogs as Presto for shared data access.

Query History page

A console view listing past queries with duration, rows processed, and stage-level details. It is the first place to diagnose slow or failed queries.

Columnar format benefit

Parquet and ORC store data by column, enabling readers to skip unused columns and compress better, which reduces I/O for analytical queries.

IAM authentication

IBM Cloud Identity and Access Management authenticates users and API calls to watsonx.data SaaS. API keys are exchanged for IAM tokens used in requests.

RBAC in watsonx.data

Role-based access control assigns roles (Admin, User) and privileges at instance, catalog, schema, or table level. GRANT and REVOKE SQL statements manage privileges.

Fine-grained access control

Presto plugins enforce column-level and row-level security. Policies can be implemented through views or custom access control rules per catalog.

IBM Knowledge Catalog integration

Publishing watsonx.data assets to IKC provides metadata enrichment, lineage tracking, and policy enforcement across the governed data estate.

Activity Tracker logging

IBM Cloud Activity Tracker records administrative actions on watsonx.data SaaS instances, supporting audit and compliance investigations with tamper-evident logs.

Engine health monitoring

Infrastructure Manager displays CPU, memory, and pod status for each engine. Alerts can be configured for failures and storage capacity thresholds.