Data Preparation and Governance with watsonx.data and watsonx.governance
This domain tests how watsonx.data assembles a lakehouse from open components, and how governance records where data and models came from.
Architecture pieces
- Storage stays in object storage (IBM Cloud Object Storage, S3-compatible, MinIO). watsonx.data adds compute and metadata, not a proprietary store.
- Engines: Presto (Trino) for interactive and federated SQL across sources; Spark for transformation and ML workloads.
- Apache Iceberg is a table format giving ACID transactions, snapshots and schema evolution. Parquet, ORC, CSV and Avro are file formats, not table formats.
- A Hive metastore tracks catalogs, schemas and tables: a catalog points at a metastore, a schema groups tables, and a bucket is registered as storage.
Ingest or virtualize
You can register existing data in place and query it, or ingest and transform into Iceberg tables. Federated queries can join Db2, PostgreSQL and object storage in one Presto statement.
Governance side
Catalogs in Cloud Pak for Data hold metadata, lineage, quality rules and business terms; watsonx.governance holds use cases, model inventory, factsheets, risk assessments and monitoring results. A factsheet pulls data and model metadata so a reviewer can trace a prediction back to its dataset.
Worked example
Parquet files sit in COS and must be queried without copying. Reasoning: register the bucket, attach a catalog and metastore, create a schema, create a table over the Parquet location, then query with Presto. Copying into Iceberg is an optimization, not a prerequisite.
Traps
Calling Parquet a table format; assuming Presto replaces Spark for writes and ETL; mixing data catalogs with the governance console; forgetting that virtualized data still depends on source permissions.