Data Integration and Interoperability
Data Integration and Interoperability
This domain tests how data moves and how systems exchange meaning, not just bytes. Expect questions on integration patterns and their trade-offs, latency and batch versus streaming, interoperability levels, and the metadata and quality controls that keep integration trustworthy.
Core rules and concepts
- DAMA defines data integration and interoperability as the acquisition, extraction, transformation, movement, delivery, replication, federation, virtualization, and operational support of data. Integration is movement and combination; interoperability is the ability of systems to exchange and use shared data.
- Patterns: ETL (transform before load), ELT (load raw and transform in the target), change data capture (read logs or triggers to move only changes), replication, federation (query distributed sources through a virtual layer), virtualization, messaging and publish/subscribe, APIs and services, and streaming.
- Batch suits large volumes and complex transformations with relaxed latency; streaming suits event-driven needs such as fraud detection. CDC reduces source impact and latency but depends on log access and schema discipline.
- Interoperability layers: technical (protocols and connectivity), syntactic (format and structure, for example XML or JSON schemas), semantic (shared meaning and code sets), and organizational (process and policy alignment). Most failures are semantic, not technical.
- Integration must carry metadata such as lineage and mappings, enforce data quality rules, secure data in motion, handle errors and reprocessing, and be monitored for freshness and completeness.
- Canonical models and standardized code sets reduce point-to-point interfaces; an enterprise service bus or API gateway can mediate, but it does not create semantic agreement by itself.
Worked example
A retailer needs store inventory visible to e-commerce within five minutes. Nightly ETL cannot meet the latency. The team chooses CDC on the inventory database, streams change events through a message broker, and applies a lightweight transformation to a canonical inventory event. A semantic mismatch appears: stores send status code A for available, while e-commerce expects AVAILABLE. A cross-reference maps the codes and the mapping is registered as metadata. The design also includes poison-message handling, replay, and monitoring for event lag. Federation was rejected because querying store systems directly would expose them to e-commerce load and would not meet the latency target reliably.
Traps that cost marks
- Treating ETL and ELT as synonyms; the transformation location changes performance and governance options.
- Choosing real-time for every requirement; latency, cost, and complexity must be justified.
- Assuming federation copies data; it queries at source and inherits source availability and security constraints.
- Ignoring semantic interoperability and calling a working API integrated.
- Building point-to-point interfaces without lineage, error handling, or a canonical model.