IDMC for Databricks Foundation study plan

Map the IDMC service modules (CDI, CAI, DQ, MDM, Data Catalog, and Data Marketplace) to the Databricks workloads each one complements, and write the mapping down

Diagram the IDMC runtime architecture end to end: Secure Agent, agent groups, the hosted runtime, and the connection layer that reaches Databricks SQL warehouses and clusters

Compare how IDMC and Databricks each represent a table, schema, and catalog, and note where the two metadata models diverge

List the Databricks-side prerequisites an IDMC ingestion pipeline needs: a SQL warehouse or job cluster, a Unity Catalog storage location, and a service principal

Stand up a working IDMC-to-Databricks connection using a service principal and a personal access token, and confirm the connection test succeeds

Build one batch mapping that reads a file source and writes to a Delta table in Unity Catalog, then run it and verify row counts match

Configure an incremental load using a watermark column and explain why a full reload would be wrong for that source

Compare the pushdown and non-pushdown execution paths and identify which transformations in your mapping force data to move through the Secure Agent

Implement a second ingestion pattern (CDC or streaming) and document the latency and ordering guarantees it actually provides

Configure schema drift handling so a new source column does not break the pipeline, and record what happens on a dropped column

Design a landing-zone layout (raw, conformed, curated) inside the Databricks lakehouse and justify the file formats at each layer

Tune one ingestion job for throughput by adjusting partitioning, batch size, or parallelism, and measure the before-and-after runtime

Build a data quality rule set in IDMC Data Quality against a Databricks table, covering at least null checks, range checks, and referential integrity

Configure a validation step that quarantines failing rows into a separate Delta table rather than dropping them silently

Explain how IDMC transformation logic maps onto Databricks SQL and Spark, and identify which transformations should be pushed down to Databricks

Set up a governance baseline: tag sensitive columns, define a data domain, and record the owner for each curated table

Catalog an ingested Databricks table in IDMC Data Catalog and verify that column-level metadata and business descriptions are populated

Trace end-to-end lineage from the source file through the IDMC mapping to the Delta table, and capture where lineage breaks

Reconcile a naming and ownership conflict between the IDMC catalog entry and the Unity Catalog entry, and document the resolution rule you adopt

Draft a glossary of at least ten business terms and link each one to a physical column

Configure role-based access in IDMC so that a developer, an operator, and a viewer each see only the objects they need

Map the IDMC permission model onto Unity Catalog grants (catalog, schema, table, and column level) and identify where the two layers must agree

Secure credentials by moving connections to a service principal or secret store, and confirm no passwords remain in plain-text connection properties

Document the audit trail: which IDMC and Databricks logs record who ran what, and how long each retains it

Deploy a pipeline from a development to a production environment using IDMC's deployment mechanism, and record every environment-specific property that had to change

Configure alerting on job failure and on SLA breach, and trigger both deliberately to prove the alerts fire

Write a runbook covering restart from a failed point, backfill of a date range, and rollback of a bad load

Measure and record baseline operational metrics: runtime, rows per run, and failure rate over several runs

Design an end-to-end solution for one realistic scenario (for example, near-real-time customer 360 or a governed finance reporting mart) covering ingestion, quality, catalog, and security

Justify each design choice against at least one alternative you rejected, and state the tradeoff explicitly

Complete two timed practice sets weighted by domain percentage, and review every missed question against its source domain

Close remaining gaps by re-reading your own notes on the two weakest domains and rewriting each weak area as a one-paragraph summary