Data Quality, Transformation, and Governance
What this domain tests
This domain covers profiling, rules, cleansing, transformation logic, and governance controls. It tests whether quality is engineered into the pipeline rather than checked after the fact.
Core rules
- Cloud Data Quality profiles data to reveal nulls, duplicates, patterns, and outliers. Rules formalize expectations: completeness, validity, uniqueness, referential integrity, and custom expressions. Scorecards summarize rule results; exception records capture failing rows.
- Cleansing standardizes and corrects values; deduplication matches and consolidates records. Run quality at ingestion, before publishing gold, and as a recurring control.
- Transformations include expression, lookup, joiner, aggregator, router, and mapplet reuse. Pushdown is appropriate only when every expression is compatible with the target engine; otherwise results can differ.
- Governance links policies, glossary terms, stewards, and catalog assets. Masking and row filters protect sensitive data; retention and access policies must be enforced where the data lives.
Worked example
A customer table has invalid emails and duplicate IDs. Profile the source, create a validity rule for email format and a uniqueness rule for customer ID. Route failing rows to an exception table with rule ID and timestamp, correct standardizable values, and publish only passing rows to silver. Log the scorecard so stewards can trend quality.
Traps
Treating profiling as enforcement; running DQ only after gold; assuming pushdown preserves every function; and confusing a glossary term with a physical column.