Working with Lakeflow Jobs
Anatomy of a job
A Lakeflow Job is a DAG of tasks. Task types include notebook, Python script or wheel, SQL query or file, pipeline (Lakeflow / Delta Live Tables), dbt, For each (loop over an array) and If/else (conditional branch). Dependencies define ordering; independent tasks run in parallel. Each task can set retries, timeouts, and parameters.
Compute and triggers
Job compute options: a dedicated job cluster created per run and terminated afterward (never reused between runs), serverless compute, or an existing SQL warehouse for SQL tasks. Triggers include scheduled (cron with timezone), continuous, file arrival (requires a Unity Catalog external location or volume), and table update. max_concurrent_runs and queueing control overlapping schedules.
Parameters
Job-level parameters reach notebooks as widgets (dbutils.widgets.get('k')) and Python scripts as argparse arguments. Task-level parameters override job-level values for that task only; dynamic references such as {{job.parameters.x}} and {{tasks.y.values.z}} pass values between tasks.
Repair and recovery
A repair run re-executes only failed tasks (or failed plus downstream) and preserves successful task outputs. It does not re-run upstream tasks that already succeeded, and it will not pick up code changes in tasks it does not re-run. If upstream logic changed, do a full run.
Traps
- Expecting a job cluster to stay warm for interactive use - it terminates.
- Assuming file-arrival triggers work on plain DBFS paths; they need UC volumes or external locations.
- Confusing widget defaults with job parameters - a missing parameter silently falls back to the default.