Data Quality and Analytics
Data Quality and Analytics tests whether you can define quality measurably, find root causes rather than patch reports, and choose analytic methods that match the question. Expect scenarios on quality dimensions, rules and scorecards, and on interpretation errors such as aggregation and sampling bias.
Core rules and concepts
- Dimensions: completeness (is it there), validity (does it conform to a rule or domain), accuracy (does it match reality), consistency (does it agree across systems), uniqueness (no duplicates), timeliness (is it current enough for use).
- Rules are testable and owned: a range check, a referential check, a freshness check, a duplicate check. Profile data first to set thresholds that reflect reality, then monitor and alert.
- Fix at the source where possible; cleansing in a downstream report hides the defect and it reappears at the next load. Root-cause categories include process, data entry, migration, interface, and system.
- Scorecards report dimension-level results per critical data element with owners and trends, not a single blended number that hides failures.
- Analytics types: descriptive (what happened), diagnostic (why), predictive (what is likely), prescriptive (what to do). Match method to question; correlation is not causation.
- Aggregation hazards: Simpson's paradox, averaging averages, and survivorship or selection bias.
Worked example
A revenue dashboard falls 12% overnight and the business blames the warehouse. Reasoning: profile the source and the load. Row counts doubled for one region, and order IDs repeat, so uniqueness failed at ingestion; a retried job was not idempotent and inserted duplicates. Completeness and validity checks passed, which is why the defect reached the dashboard. The fix belongs in the source pipeline: key the load on order ID with a merge, add a duplicate-count rule as a blocking check, and reconcile control totals against the source each run. Report-level deduplication is temporary only. Note the dimensions involved: uniqueness (failed), timeliness (job delay), consistency (dashboard versus ledger).
Traps that cost marks
- Confusing validity with accuracy: a value can pass a format rule and still be wrong.
- Cleansing in the BI layer instead of preventing recurrence.
- Reporting one blended quality score with no owner or dimension detail.
- Treating a moving average or an average of averages as the true metric.
- Drawing causal conclusions from an observational dashboard.