Data Quality
Data Quality
Data Quality questions test dimensions, measurement, root-cause analysis, and control. The examiner wants to see that you can define a metric precisely, profile data before cleansing it, and build controls that prevent recurrence rather than running one-off cleanups.
Core rules and concepts
- DAMA core dimensions: completeness, timeliness, validity, accuracy, consistency, and uniqueness. Related characteristics include precision, accessibility, conformance, integrity, and reasonableness.
- Each dimension needs an operational definition, a business rule, a threshold, and a measurement method. Saying email is complete is not a metric; percentage of active customer records with a non-null, syntactically valid email at month end is.
- Data profiling establishes the baseline: null counts, distinct counts, min and max, frequency distributions, pattern analysis, and foreign-key orphan checks. Profile before cleansing so improvement can be proven.
- Root-cause analysis distinguishes data-entry errors, integration defects, missing business rules, and process gaps. Use the five whys, cause-and-effect diagrams, and Pareto analysis on issue types.
- Data quality management is a cycle: define quality requirements and rules, assess against them, improve through remediation and process change, and monitor continuously with scorecards and issue logs.
- Cost of poor quality includes rework, failed regulatory reporting, lost revenue, and mistrust; quantify it in business terms to justify controls.
Worked example
A customer table has 1,000 active records. 60 have no email; 25 of the 940 non-null emails fail the format rule. Completeness of email is 940 divided by 1,000, or 94 percent. Validity is 915 divided by 1,000, or 91.5 percent, if measured against all active records, or 915 divided by 940, or 97.3 percent, if measured only among populated values. State the denominator explicitly, because the two figures drive different conclusions. Profiling shows most invalid values arrive from a legacy batch feed, while missing values cluster in records created by a mobile app whose form does not require email. Root cause is therefore twofold: a feed mapping defect and a capture-process rule gap. Remediation is a mapping fix, a form validation rule, and a monthly monitor with an agreed threshold.
Traps that cost marks
- Reporting a single quality score that hides which dimension failed.
- Confusing accuracy (matches reality) with validity (conforms to an allowed format or domain).
- Cleansing data without profiling or root-cause analysis, so defects return.
- Measuring timeliness as load duration rather than data availability against a business deadline.
- Omitting data quality roles and issue escalation; quality needs ownership, not just tools.