IBM watsonx Data Lakehouse Engineer Associate key terms

IBM's open lakehouse service that combines object storage, open table formats, and multiple query engines so teams can run analytics and AI workloads on data without moving it into a single proprietary warehouse.

An architecture that stores data in low-cost object storage while adding table management, schema, and transactional guarantees, letting BI and machine learning read the same governed data with warehouse-like reliability.

An open table format that tracks table snapshots and metadata files, enabling ACID transactions, schema evolution, partition evolution, and time travel across engines reading the same underlying data files.

A service that records which tables exist, where their metadata lives, and which schema they use, so each query engine can resolve a table name to its current Iceberg or Hive metadata.

A catalog service originally built for Hive that stores database, table, and partition definitions; in a lakehouse it commonly serves as the shared catalog other engines consult.

A distributed SQL query engine in watsonx.data used mainly for interactive, federated queries across lakehouse tables and connected external sources, returning results without copying data.

A distributed processing engine in watsonx.data suited to large batch transformations, ETL, and machine learning workloads, often writing curated results back as Iceberg tables.

The compute component that parses SQL or jobs, plans execution, and reads data files from storage; watsonx.data lets several engines share the same tables and catalog.

Querying data held in external systems through a connector so results combine with lakehouse tables in one statement, avoiding a full copy into the lakehouse first.

A container in cloud object storage such as IBM Cloud Object Storage or Amazon S3 that holds data and metadata files; watsonx.data registers buckets as storage locations for catalogs.

The provisioning step of telling watsonx.data which object storage bucket to use, supplying endpoint and credentials so engines can read and write table files there.

Creating and sizing a watsonx.data deployment, including choosing a cloud region, compute sizes for engines, and storage associations before any data is ingested.

Adding or removing engine compute units, or pausing idle engines, to match workload demand and control cost while keeping the same catalog and storage intact.

A saved definition of how watsonx.data reaches an external system, holding host, port, database name, and credentials so ingestion jobs or federated queries can use it.

A configured flow that reads data from a source connection and writes it into a lakehouse table, optionally applying schema mapping, filtering, and scheduling.

The step of defining or detecting column names and data types for incoming data so it can be written as a queryable table with consistent structure.

A credential pair used to authenticate programmatic access to object storage; watsonx.data stores it securely and uses it when engines read or write bucket files.

A protected store that holds connection secrets such as passwords and storage keys, letting jobs and engines reference them without exposing plaintext values in configuration or code.

Granting permissions by assigning users or groups to roles, so who can view catalogs, query tables, or administer engines is decided by role rather than individual settings.

Restricting which users or groups may see or query a specific catalog, database, or table, providing finer granularity than broad instance-wide roles.

Recording who ran which query or administrative action and when, producing evidence for compliance reviews and helping investigate unexpected data access.

Observing running and completed queries for duration, state, resource use, and errors, so slow or failing workloads can be diagnosed and corrected.

A retained record of previously submitted statements with their status, runtime, and submitter, useful for auditing usage patterns and reproducing a past result.

Adjusting engine size, partitioning, file layout, and query structure to reduce scanned data and runtime for recurring analytical workloads.

Skipping data files whose partition values cannot satisfy a query's filters, cutting the amount of storage read and usually shortening response time.

Rewriting many small data files into fewer larger ones so query planning and reads do less overhead work, which matters after frequent small ingestion batches.

Each write to an Iceberg table creates an immutable snapshot; time travel queries an earlier snapshot, supporting reproducibility and recovery from bad loads.

An Iceberg metadata file listing the data files that belong to a table snapshot, along with statistics that let the planner skip files.

Using stored per-file statistics such as minimum and maximum values to avoid reading files that cannot contain matching rows, reducing I/O for selective queries.

A table whose definition lives in the catalog while its data stays in files at a storage location, so the data can be queried without being loaded into managed storage.

A lakehouse table whose data files and metadata are both created and maintained by the platform, so dropping the table also removes its underlying files.

Separating engines or compute resources by team or workload so a heavy batch job cannot starve interactive queries of CPU and memory.

A record of where data came from and how it was transformed, letting engineers trace a table back to its sources and assess the impact of a change.

The policies and controls that determine who may access data, how quality is maintained, and how usage is documented across the lakehouse.