Data Quality and Integrity
What the exam tests
Data quality and integrity tests definitions of quality dimensions, profiling, validation, reconciliation, provenance, and audit trails. You must distinguish accuracy from completeness and detect duplication, inconsistency, and timeliness problems.
Core rules
- Common dimensions: accuracy, completeness, consistency, timeliness, validity, uniqueness, and integrity.
- Data profiling summarizes nulls, ranges, patterns, duplicates, and referential integrity before use.
- Edit checks prevent or detect invalid entries; they must be validated and monitored.
- ALCOA+ and audit trails preserve attributable changes and original values.
- Root-cause correction is preferred over repeated manual cleaning.
Worked example
A registry has duplicate MRNs. Profiling shows exact duplicates and near-duplicates with transposed birth dates. Do not delete records outright. Define a matching rule, review candidates, merge with a documented survivorship rule, preserve the original identifiers and audit trail, and correct the registration workflow that allowed duplicates. Reconcile counts before and after. The reasoning: merging without provenance destroys integrity, while fixing only the output leaves the cause.
Traps
- Equating a complete field with a correct field.
- Cleaning source data without documenting transformations.
- Deleting apparent duplicates before checking legitimate repeat encounters.
- Ignoring timeliness when data are accurate but late.