CDDP cheat sheet

Data governance is the decision-rights and accountability framework for data assets; it defines who decides, who executes, and who is accountable, distinct from day-to-day data management.

A RACI matrix assigns exactly one Accountable per activity; multiple Responsible parties are allowed, but multiple Accountable parties create decision deadlock.

A data steward owns domain-level quality and definitions operationally; a data owner holds budgetary and policy accountability, usually at executive level.

The DAMA-DMBOK governance context includes six core activities: strategy, policy, standards, oversight, projects, and issue management.

A data policy is a directive statement of principles; a standard is a mandatory rule with measurable criteria; a guideline is a recommended practice with no enforcement.

Data ethics principles commonly cited: fairness, transparency, accountability, privacy, beneficence, and non-maleficence; ethical review precedes technical implementation.

A data governance council (steering committee) sets cross-domain policy; domain councils resolve conflicts within a subject area before escalation.

Data lineage and a business glossary are governance instruments: the glossary fixes term definitions, lineage fixes provenance and transformation accountability.

KPIs for governance maturity include policy coverage, glossary term adoption rate, issue resolution cycle time, and percentage of critical data elements with assigned owners.

Three-schema architecture: conceptual (business concepts, technology-neutral), logical (attributes, keys, relationships, still DBMS-neutral), physical (tables, indexes, partitions, storage).

In Chen notation, a diamond is a relationship, a rectangle an entity, an ellipse an attribute, and a double ellipse a multivalued attribute; crow's foot notation marks cardinality at the line end.

First normal form requires atomic attribute values and no repeating groups; second removes partial dependencies on part of a composite key; third removes transitive dependencies on non-key attributes.

A star schema has one fact table with foreign keys to denormalized dimension tables; a snowflake schema normalizes dimensions into hierarchies, trading join cost for storage savings.

Slowly changing dimension Type 1 overwrites history; Type 2 adds a new row with surrogate key and validity dates; Type 3 adds a previous-value column for limited history.

A data vault model uses hubs (business keys), links (relationships), and satellites (descriptive attributes with load timestamps), giving auditable, insert-only, parallel-loadable structures.

Kimball bottom-up builds conformed marts with bus-matrix shared dimensions; Inmon top-down builds a normalized enterprise warehouse feeding dependent marts.

A surrogate key is a system-generated integer with no business meaning; a natural key is a real-world identifier and may change, so it is stored as a business key attribute.

Lambda architecture runs batch and speed layers merged by a serving layer; Kappa architecture drops the batch layer and reprocesses everything through the stream.

ACID: atomicity (all-or-nothing), consistency (constraints preserved), isolation (concurrent transactions do not interfere), durability (committed data survives failure).

BASE trades strong consistency for availability: basically available, soft state, eventual consistency, typical of distributed NoSQL stores.

CAP theorem: under a network partition a distributed store must choose consistency or availability; partition tolerance is not optional in distributed systems.

ETL transforms before loading into the warehouse; ELT loads raw data first and transforms inside the target, common with columnar cloud warehouses.

Change data capture reads the database log or triggers to propagate inserts, updates, and deletes incrementally, avoiding full reloads.

Idempotent loads can be rerun without duplicating rows; achieve via MERGE/upsert on a business key, truncate-and-reload partitions, or deduplication keys.

Row-oriented storage suits OLTP point lookups; columnar storage with compression and zone maps suits OLAP scans and aggregation.

A data lake stores raw files in open formats with schema-on-read; a lakehouse adds ACID table formats (Delta, Iceberg, Hudi) and catalog metadata over object storage.

Integration patterns include point-to-point, hub-and-spoke, enterprise service bus, and publish-subscribe messaging; each adds coupling or latency trade-offs.

Six core data quality dimensions: completeness, accuracy, consistency, timeliness, uniqueness, and validity (conformity to format or domain rules).

A data quality rule is testable and specific, e.g. 'customer_email matches RFC 5322 pattern for 99.5% of active rows'; a vague goal is not a rule.

Data profiling measures column statistics (null rate, distinct count, min/max, pattern frequency) before rules are written, to discover actual defects.

A critical data element is an attribute with material impact on reporting, regulatory compliance, or customer experience; CDEs get mandatory stewardship.

MDM styles: registry (index only, no golden record), consolidation (golden record for reporting), coexistence (bidirectional sync), and central (transactional system of entry).

A golden record is the trusted, survivorship-ruled version of a master entity; survivorship rules pick source values by recency, completeness, or trust ranking.

Entity resolution matches records across sources via deterministic keys then probabilistic scoring (e.g. Fellegi-Sunter), followed by merge and clerical review.

Data quality issue root causes fall into process, system, and people categories; a control must be added at the point of creation, not only detected downstream.

A DQ scorecard reports dimension-level pass rates per dataset over time; thresholds trigger alerts and remediation workflows.

Mean is sensitive to outliers; median is robust; for skewed distributions report median and interquartile range rather than mean and standard deviation.

Standard deviation measures spread in original units; variance is its square; coefficient of variation = standard deviation / mean, enabling spread comparison across scales.

Central limit theorem: sampling distribution of the mean approaches normal as n grows, regardless of population shape, enabling confidence intervals and t-tests.

p-value is the probability of data at least as extreme given the null hypothesis is true; it is not the probability the null is true.

Type I error is rejecting a true null (false positive, rate alpha); Type II is failing to reject a false null (false negative, rate beta); power = 1 - beta.

Correlation coefficient r ranges from -1 to 1 and measures linear association only; correlation does not establish causation without design or experiment.

Simple linear regression: y = a + bx, where b = covariance(x,y)/variance(x); R-squared is the proportion of variance in y explained by the model.

Bayes' theorem: P(A|B) = P(B|A)·P(A)/P(B); base rates dominate posterior probability when the test is applied to a rare condition.

Sampling methods: simple random, stratified (proportional to subgroup), cluster, and systematic; selection bias invalidates inference regardless of sample size.

Encoding accuracy ranking: position on a common scale > length > angle or slope > area > color hue; use position for the most important quantitative comparison.

Bar charts compare categorical magnitudes; line charts show change over ordered time; scatterplots show relationship between two measures; pie charts only for few parts of one whole.

Truncated y-axis on a bar chart exaggerates differences because bar length encodes value; line charts may truncate when change, not magnitude, is the message.

Data-ink ratio (Tufte): maximize ink used for data, remove chartjunk, gridlines, and redundant labels; small multiples replace decorative complexity.

Pre-attentive attributes (color, size, position, orientation) are perceived in under 250 ms; use one or two to highlight, not all at once.

Colorblind-safe palettes avoid red-green discrimination; roughly 8% of men have some color vision deficiency, so pair color with shape or label.

Dashboard hierarchy: overview first, then zoom and filter, then details on demand (Shneiderman's mantra); each KPI needs a defined owner, formula, and refresh cadence.

Accessibility requires text alternatives, minimum contrast ratio 4.5:1 for normal text, and no information conveyed by color alone.

CIA triad: confidentiality (authorized access only), integrity (data unaltered and trustworthy), availability (accessible when needed); each control maps to at least one.

Authentication verifies identity; authorization grants permissions; accounting/auditing records what was done; all three are needed for access control.

Encryption at rest protects stored media; encryption in transit (TLS) protects network paths; key management is the weakest link in both.

Tokenization replaces a value with a non-sensitive token mapped in a vault; format-preserving encryption keeps structure but remains reversible with a key.

Data masking (static or dynamic) obscures values for non-production or role-restricted use while preserving referential and format properties.

GDPR principles: lawfulness, fairness, transparency, purpose limitation, data minimization, accuracy, storage limitation, integrity and confidentiality, accountability.

A data subject access request must be fulfilled within one month under GDPR, extendable by two further months for complex requests.

Data classification tiers (public, internal, confidential, restricted) drive handling rules; PII and PHI are regulatory categories requiring stricter controls.

Least privilege and role-based access control limit blast radius; segregation of duties prevents one identity from both changing and approving a sensitive action.

Object storage (S3-compatible) offers high durability and cheap capacity with high latency; it is the default landing zone for lake and lakehouse architectures.

MPP warehouses distribute and parallelize queries across nodes; distribution key choice determines data shuffle cost on large joins.

Streaming platforms (Kafka-style) model topics, partitions, and consumer groups; ordering is guaranteed within a partition, not across partitions.

Exactly-once processing is achieved via idempotent writes plus checkpointed offsets; at-least-once delivery with deduplication is the common practical pattern.

Open table formats add ACID transactions, time travel, and schema evolution over files: Delta Lake (transaction log), Apache Iceberg (snapshots and manifests), Apache Hudi (timeline).

Data mesh is an organizational pattern: domain ownership, data as a product, self-serve platform, and federated computational governance; it is not a product you buy.

ML pipelines require feature stores for training-serving consistency and model monitoring for drift; data drift and concept drift differ in whether the target relationship changes.

Vector databases index embeddings for similarity search using approximate nearest neighbor methods (HNSW, IVF); recall and latency trade off against index size.

Serverless and elastic compute decouple storage from compute, so cost scales with query duration; unpartitioned scans dominate cloud warehouse spend.