Data Ingestion and Connection Management
This domain tests connections and ingestion: how watsonx.data reaches sources, how data lands in lakehouse tables, and how job settings affect correctness and repeatability.
Two ways data appears
- Register external data: create an external table over files already in object storage, or query files in place. Metadata is created; data is not moved.
- Ingest data: run an ingestion job that reads a source and writes a target table, usually Parquet or ORC in an Iceberg catalog.
Connections hold endpoint, port, database, and credentials; always test before saving. Ingestion jobs define source, target catalog/schema/table, file format, write mode, schedule, and column mapping.
Worked example
A nightly job copies an Oracle SALES table into Iceberg. Create an Oracle connection with host, port, service name, and a read-only account; test it. Create an ingestion job targeting sales_lake.raw.sales, format Parquet, mode overwrite, schedule daily. Overwrite is correct because each run is a full snapshot; append would duplicate every row. The target table is partitioned by sale_date so later queries prune files.
Traps
- Using append for a full snapshot, then wondering why counts double.
- Giving ingestion a source account that cannot read the table.
- Confusing external tables (no copy) with ingestion (copy).
- Ignoring column mapping and data types; a string landing in a date column fails or nulls.
- Forgetting that file format and partitioning are decided at target creation.