Data Quality Management
Data quality management defines fitness for purpose, measures it, finds root causes, and sustains remediation. The exam tests quality dimensions, profiling, rule design, issue triage, and how quality is governed and reported.
Core rules
- Common dimensions include completeness, accuracy, validity, uniqueness, timeliness, consistency, and integrity. Select dimensions by use case; “quality” without a purpose is untestable.
- Profile before writing rules: measure null rates, distinct counts, patterns, ranges, and duplicate rates to establish a baseline.
- Rules must be testable and owned: define the rule, threshold, scope, frequency, and remediation path.
- Separate detection from correction. Root-cause analysis asks where the defect entered - entry, interface, transformation, or reference data - before cleansing downstream.
- Report quality with trend, impact, and exceptions; a scorecard that never triggers action is theater.
Worked example
A customer file shows 2.3% duplicate records by tax ID. Profiling finds the duplicates arise when the web form allows a second registration before the overnight load. The team adds a match rule at entry, a daily duplicate metric with a 0.5% threshold, and an exception queue for legitimate shared tax IDs. After four weeks the rate falls to 0.2%; the steward reports the trend and the remaining exceptions. The reasoning links measurement to the entry-point cause rather than repeatedly cleansing the warehouse.
Traps
- Treating data quality as an IT cleanup project with no business owner.
- Setting thresholds with no baseline or impact rationale.
- Over-fitting rules to one extract and missing broader patterns.
- Cleansing data without fixing the source, so defects return.