Use Cases and Solution Design on Databricks
What this domain tests
Use Cases and Solution Design tests architecture judgment. You must match business requirements to latency, transformation placement, governance, and operating model.
Core rules
- Start with the SLA and data shape. Batch suits reporting; CDC or streaming suits operational freshness. Choose where transformation runs based on skills, cost, and reuse: IDMC for source-side cleansing and reusable mappings, Databricks for large-scale SQL/Spark modeling.
- A common reference architecture: sources -> IDMC ingestion and quality gates -> bronze Delta -> Databricks silver/gold transformations -> BI and reverse ETL. Unity Catalog and Metadata Command Center provide access control, catalog, and lineage.
- Design for failure and change: idempotent loads, schema evolution, replay, and clear ownership. Avoid duplicating the same transformation in both platforms.
- Consider licensing and compute: IDMC runtime capacity, Databricks cluster or warehouse size, and storage growth all affect total cost.
Worked example
A retailer wants customer 360 in 30 minutes. Ingest CRM, e-commerce, and POS changes with IDMC CDC into bronze. Apply email and customer-ID quality rules at ingestion. Databricks deduplicates and builds a silver customer hub, then a gold segment table. Unity Catalog restricts PII, and lineage links the gold segment back to source systems for impact analysis.
Traps
Choosing streaming for a daily report; transforming everything in IDMC and losing lakehouse scalability; ignoring governance until after launch; and designing without an idempotent replay path.