Monitoring and Alerting
Where the signal lives
Match the symptom to the observability source - that is the whole game in this domain. Job-level failures and durations: job run history and notifications. Pipeline quality: the Lakeflow pipeline event log. Cluster resource issues (spill, GC, skew): the Spark UI and its stages/SQL tabs. Streaming lag and throughput: query.lastProgress, query.status, and StreamingQueryListener metrics such as numInputRows, inputRowsPerSecond, processedRowsPerSecond. Cost and usage: system.billing.usage. Access and permission anomalies: system.access.audit. Lineage: system.access.table_lineage.
Alerting mechanics
Job notifications can fire on start, success, failure, or duration threshold, to email or to a webhook destination (Slack, Teams, PagerDuty). Databricks SQL alerts run a query on a schedule and trigger when a condition is met, with retries and optional subscriptions. Query history and query profiles explain slow dashboards.
Trap: alerts on streaming tables/queries do not replace checkpoint-based health monitoring; a query can be running with zero input rows and look "successful". Also, deleting a checkpoint location silently restarts from the source rather than resuming.
Rule of thumb: instrument outputs, not jobs - assert on row counts/freshness in a quality table, and alert on that table rather than on whether the job "succeeded".
Keep retention in mind: audit logs default to 90 days in system tables, so long-trend investigations need export.