IDMC for Databricks Foundation cheat sheet

Use Cases and Solution Design on Databricks

• Pattern selection: batch file ingestion → Auto Loader or COPY INTO; database replication → CDC with MERGE or AUTO CDC; SaaS APIs → Lakeflow Connect or Mass Ingestion connectors.

• When to keep IDMC in the architecture: heterogeneous multi-source integration, complex data quality and matching, business glossary and governance requirements, or existing Informatica investment.

• When to push down to Databricks: large-volume joins and aggregations where transformations are pushdown-eligible, minimizing data movement and Secure Agent load.

• Medallion design rule: Bronze preserves source fidelity and history; Silver applies cleansing, deduplication, and conformance; Gold serves curated marts and ML feature tables.

• Streaming design: Kafka/Event Hubs → Bronze streaming table → expectations in Silver → Gold aggregation; use checkpointing, watermarks, and idempotent MERGE for exactly-once semantics.

• Data quality gating: run profiling and rules before promotion to Silver; failing thresholds block the pipeline via expect_or_fail or an orchestrated gate.

• Governance-first design: register assets in Unity Catalog, tag PII, apply column masks and row filters, and publish glossary terms in CDGC for discoverability.

• ML and analytics enablement: Gold tables feed Databricks SQL dashboards and MLflow experiments; feature tables should be governed and versioned in Unity Catalog.