Data integration and virtualization
Integration questions ask you to choose the right movement strategy and size it.
Patterns
- ETL/ELT with DataStage: transform and persist into a target; best when data must be reshaped, historized, or served from a warehouse.
- Replication with Data Replication (CDC): low-latency copies of source changes; best for operational reporting and offloading production.
- Virtualization with Watson Query (called Data Virtualization before 4.6): query sources in place through virtual tables; best when data cannot move, is distributed but fresh, and query volumes are moderate.
- Notebooks/Spark in Watson Studio for exploration and feature preparation.
Watson Query mechanics
Connections feed virtualized tables, which can be joined and optionally cached. The engine pushes predicates and joins to sources where possible; complex cross-source joins run in the engine. Caching holds result sets in engine storage, trading freshness for latency. Grants are applied to virtual tables, not to the source objects.
Worked example
A reporting team needs one SQL view over on-premises Db2 and S3 Parquet, refreshed hourly, for ten analysts. Virtualize both sources, join in Watson Query, cache the small Db2 dimension table rather than the fact table, and schedule an hourly refresh that matches the SLA. If the SLA were sub-second over billions of rows, replicate to Db2 Warehouse instead - virtualization would push too much work onto the source.
Traps
Assuming virtualization copies data; ignoring source-side CPU and concurrency limits; caching a large volatile table and serving stale rows; forgetting that source credentials live in a shared connection used by every virtual table built on it.