Data Quality and Integration
What to know
Data quality is fitness for use, measured across dimensions: accuracy, completeness, consistency, timeliness, validity, uniqueness, and integrity. Candidates must know the difference between dimension (what you measure) and metric (the threshold/score). Understand the quality cycle: profile → assess → remediate at source → monitor → govern. Root-cause remediation at the point of creation beats downstream cleansing.
Integration covers ETL vs. ELT (transform before vs. after load), batch vs. real-time/streaming, and patterns: consolidation, federation, and propagation. Know the concepts of data lineage, provenance, and reconciliation. Understand matching and deduplication (deterministic vs. probabilistic), and master data management's role in resolving entity conflicts. Familiarity with data virtualization and APIs as integration alternatives to physical copying is useful.
Traps
- Treating data cleansing as a permanent fix rather than a symptom treatment - fix the source.
- Confusing completeness (no missing values) with accuracy (values are correct).
- Assuming ETL and ELT differ only in tooling - the transform timing changes governance and performance trade-offs.
- Forgetting that integration introduces latency and lineage complexity.
Worked rule of thumb
When a quality issue appears, ask three questions in order: Where is it created? Who owns that source? What control prevents recurrence? This routes you to root cause and stewardship rather than a cleansing band-aid.
For matching, deterministic rules (exact keys) are precise but brittle; probabilistic matching (e.g., similarity scoring) catches variants but needs tuning and human review for high-stakes merges.