Cloud Data Quality R41 Professional key terms

A named, measurable aspect of data fitness such as completeness, accuracy, uniqueness, timeliness, validity, or consistency. Rules and scores are usually organized around these dimensions so teams can report quality in a comparable way.

A declarative test that data must satisfy, such as a null check, range check, or referential check. Each rule returns pass or fail counts and can be scheduled, versioned, and tied to an alert.

The process of scanning a dataset to compute statistics such as null rates, distinct counts, minimum and maximum values, and value distributions. Profiling results inform which rules to write and what thresholds are realistic.

Profiling focused on a single field, producing metrics like null percentage, distinct value count, top values, and inferred data type. It is usually the first step before writing column-level validation rules.

The degree to which required values are present in a dataset. It is commonly measured as the percentage of non-null values in mandatory fields or as the share of expected records actually loaded.

The property that a key or combination of keys appears at most once in a dataset. It is tested with duplicate checks and is essential for primary keys and slowly changing dimensions.

A constraint ensuring that values in a foreign key column exist in the referenced parent table. Violations reveal orphan records caused by late-arriving, deleted, or mis-keyed source data.

A child row whose foreign key has no matching parent row. Orphans typically arise from out-of-order loads or deleted reference data and are detected by referential integrity checks.

A rule verifying that a column contains no missing values, or that the null rate stays below a threshold. It is the most common completeness test and is applied to mandatory fields.

A rule that counts rows sharing the same key values and flags groups larger than one. Results distinguish exact duplicates from business-key duplicates that differ only in non-key attributes.

A rule confirming that values conform to an expected type, such as integer, date, or decimal. Type mismatches often indicate upstream schema drift or badly formatted source extracts.

A validation that values conform to a regular expression or format, such as a postal code or email layout. It catches structurally invalid values that pass type checks but are still unusable.

A rule asserting that numeric or date values fall within expected bounds. It detects impossible values such as negative ages or future birth dates and is often paired with distribution analysis.

A configured limit that determines when a rule result counts as a failure, expressed as an absolute count or a percentage. Thresholds balance sensitivity against alert noise in production monitoring.

A numeric summary of rule outcomes, often a weighted percentage of passing checks across dimensions. Scores let teams track trends over time and compare datasets or domains.

Automatic identification of unusual data patterns, such as sudden null spikes or volume drops, without a hand-written rule. Machine-learned baselines adapt to seasonality and reduce manual threshold tuning.

A measure of how recently a dataset was updated relative to expectations. Stale data is detected by comparing the latest load timestamp or maximum event time against a service-level expectation.

A rule comparing the number of rows loaded in a period against an expected range or historical baseline. It quickly catches failed or partial loads before downstream consumers are affected.

A visual interface showing rule pass rates, score trends, and failing records across datasets. It supports triage by letting stewards drill into specific incidents and affected tables.

The mechanism that notifies owners when a rule fails or a threshold is breached, via email, chat, or ticketing integrations. Good alerting routes issues to the right owner with context and severity.

A tracked record of a detected quality failure, including scope, severity, root cause, and resolution. Incident workflows connect detection to remediation and provide an audit trail.

The practice of tracing a quality failure back to its origin, such as a source schema change or a faulty transformation. It prevents recurrence instead of only fixing the visible symptom.

A record of how data flows from sources through transformations to downstream tables and reports. Lineage enables impact analysis when a failure occurs and helps locate the upstream cause.

Determining which downstream datasets, dashboards, and consumers are affected by a data defect or schema change. It uses lineage and dependency metadata to prioritize remediation effort.

An unannounced change to the structure of source data, such as a renamed or retyped column. It commonly breaks pipelines and validation rules and is best caught by schema comparison checks.

A person accountable for the quality and meaning of a data domain, who defines rules, reviews incidents, and approves fixes. Stewards bridge business definitions and technical implementation.

A searchable inventory of datasets with descriptions, owners, schemas, and quality metadata. It helps consumers find trusted data and gives stewards a place to publish rule results.

A curated list of agreed business terms and their definitions, linked to physical data assets. It reduces ambiguity in rule design by giving metrics and fields a single shared meaning.

A documented commitment about data delivery, such as freshness, completeness, or availability targets. Quality rules operationalize these commitments by testing them continuously.

A separate storage location where rows failing validation are written instead of being discarded. It preserves bad records for investigation and allows reprocessing after correction.

A holding area for messages or records that a streaming pipeline could not process. It prevents one bad record from halting a pipeline and supports later inspection and replay.

The actions taken to correct data defects and restore trust, including cleansing, reprocessing, and rule updates. Effective remediation also documents the fix and verifies that scores recover.

Applying quality checks to records in motion, before or as they land, rather than in batch later. It enables near-real-time rejection or routing but demands low-latency rules and careful state handling.

A value that deviates markedly from the rest of its distribution. Outliers may be genuine rare events or errors, so statistical detection should be paired with business context before rejection.

Records that reach the warehouse after their expected processing window, often from delayed sources. They complicate referential checks and aggregations and may require reprocessing or backfilling.

One or more columns that uniquely identify each row in a table. Validating primary key uniqueness and non-nullness is a foundational quality check for every curated dataset.

A column referencing the primary key of another table to express a relationship. Foreign keys underpin referential integrity checks that detect orphaned child records.

The degree to which values conform to defined formats, domains, or business rules. It differs from accuracy, which concerns whether values match reality.