Working with Lakeflow Jobs
Working with Lakeflow Jobs
This domain tests orchestration: building multi-task jobs, choosing triggers and compute, passing parameters, handling failures, and operating runs. Expect questions about task dependencies and production job behavior.
Job design rules
- A Lakeflow job is a DAG of tasks. Task types include notebook, Python script, Python wheel, SQL, pipeline, dbt, run job, and control-flow tasks such as If/else and For each. Dependencies define the DAG; tasks without dependencies can run in parallel.
- Use job clusters or serverless job compute for scheduled work; all-purpose clusters are for interactive development. A SQL task needs a SQL warehouse.
- Triggers: scheduled cron with a timezone, file arrival, continuous, or manual. Paused schedules keep configuration but do not run.
- Parameters: job parameters flow to tasks; notebooks receive widgets, Python scripts use
argparse/dbutils.widgets, and dynamic references such as{{job.id}}and{{task.name}}are available in fields. - Reliability: set retries and timeouts per task, configure notifications, and use repair run to rerun only failed tasks. Use run-as a service principal for production ownership.
- Concurrency: set max concurrent runs to prevent overlapping schedules; use queueing or skip behavior deliberately.
Worked example
A nightly pipeline ingests files, transforms silver, checks quality, and refreshes a dashboard. It must start when files arrive and not rerun successful ingestion after a transform failure. Reasoning: create four dependent tasks and a file-arrival trigger. Give the transform task a retry and a timeout. If the quality task fails, use repair run so ingestion is not repeated. Run as a service principal, use a job cluster, and set a maximum concurrent runs limit of 1 to avoid overlapping file batches.
Traps
- Scheduling jobs on an all-purpose cluster.
- Hardcoding paths and dates instead of job/task parameters.
- Using a personal user as run-as for production.
- Repairing by rerunning the whole job when only one task failed.
- Ignoring timezone in cron schedules.
- Letting a file-arrival trigger fire multiple runs without a concurrency limit.