Deployment, Monitoring, and Operations
What this domain tests
Deployment, Monitoring, and Operations tests the run lifecycle: promotion, scheduling, observability, recovery, and cost control. Questions often describe a failure and ask for the first diagnostic step.
Core rules
- Promote assets through dev, test, and production with parameterized connections and schedules. Export/import or repository promotion must carry dependencies; verify that connection and runtime names are environment-specific.
- Monitor at three layers: IDMC task status and logs, Secure Agent or serverless runtime health, and Databricks job runs, query history, and cluster metrics. Configure alerts for failures and SLA breaches.
- Design for idempotency: watermarks, merge keys, and deduplication prevent double loads on retry. Handle schema drift with explicit evolution rules.
- Cost and capacity: right-size clusters and warehouses, avoid idle all-purpose clusters, control small files, and schedule heavy loads off peak.
Worked example
A nightly load fails at 02:00. Check the IDMC task log first: if it shows a connection timeout, inspect the Secure Agent heartbeat and network path. If the extract succeeded but the Databricks write failed, check the job run, Unity Catalog grants, and Delta commit conflict. Fix the cause, then rerun with the same watermark and MERGE to avoid duplicates.
Traps
Rerunning an append load without deduplication; monitoring only Databricks and not the agent; hard-coding production connection names; and ignoring small-file and idle-compute cost.