Databricks Data Engineer: Repair Runs, Spot Loss, and Cluster Topology in a Nine-Task Job

Repair Runs, Spot Loss, and Cluster Topology in a Nine-Task Job

A nightly Lakeflow job has nine tasks: a Python wheel ingestion task, four silver notebooks that can run in parallel, a SQL gold task, a validation task, and a publish task. Everything runs on one autoscaling job cluster (8 workers). The run takes 3h40m. When something fails, the on-call engineer clicks Repair run with all tasks selected, which re-runs ingestion and all four silver tasks. Last month spot reclamation killed two silver tasks mid-run; task retries are set to 0, so the whole job was re-run and finished after the 06:00 SLA. Ingestion writes into bronze with INSERT INTO from a batch source and is not safe to execute twice.

1. How do you shrink the blast radius of a failure without weakening the pipeline's guarantees?

Read the worked answer

Three changes, in order of impact:

Task-level retries. Set max_retries and min_retry_interval_millis on the silver tasks so a transient spot reclamation is absorbed inside the run. Retries are per task, so the ingestion task is unaffected.
Make ingestion idempotent. Because INSERT INTO from a batch source duplicates on re-run, ingestion should be a MERGE on a natural key or a partition overwrite of the logical window. Until that is true, ingestion should never be included in a repair, which forces the engineer to think about it every time.
Repair only what broke. Repair run lets you select a subset of tasks; selecting the failed tasks and their downstream dependents re-runs exactly those and leaves ingestion alone. Selecting all tasks is what turned a two-task failure into a full 3h40m rerun.

Additionally, gate publish on validation with a Run if / condition dependency, so a failed validation never reaches consumers and a retried validation can succeed without touching ingestion.

2. The four silver tasks run in parallel on the shared 8-worker cluster and often queue. How should compute be laid out?

Read the worked answer

A single shared job cluster serializes the parallel tasks: four notebooks contending for 8 worker slots means each waits for the others, so the wall clock looks like the sum rather than the max. The 3h40m run is likely dominated by that queueing plus repeated spills.

Options, roughly in order of preference:

Split into a few compute keys. Put the four silver tasks on their own cluster (or two clusters of two tasks each) and keep the light tasks on a shared cluster. Tasks with different compute keys launch in parallel, which is the point of the DAG. The cost is several minutes of cluster start-up per group, which is trivial against a 3h40m run.
Keep one cluster but size it deliberately. If cluster sprawl is unacceptable, make the shared cluster large enough for the real concurrency instead of relying on autoscaling to catch up mid-run; autoscaling reacts in minutes, not seconds.

Also reconsider spot for these tasks: spot workers are fine only when retries are configured and tasks are resilient to a restart, and the driver should not be on spot. Given the incident history, that pairing is the actual fix.

3. The SLA is 06:00 and the job starts at 01:00. Where do you look to find the time, and what scheduling settings matter?

Read the worked answer

Start with the run's task timeline in the job run UI, which shows per-task duration and where parallel branches actually overlap versus queue. That immediately tells you whether the critical path is ingestion, one slow silver notebook, or the queueing described above. For history, query the Lakeflow system tables (for example job and task run timeline tables) to rank tasks by p95 duration over the last 30 days rather than reasoning from a single run.

Scheduling settings that matter here:

max_concurrent_runs. If a run overruns, the next scheduled run may start on top of it and both compete for the same cluster and tables. Cap concurrency at 1 and let the schedule queue, so overruns degrade to lateness rather than to concurrent writes.
Timeout and notification. Set a task or run timeout below the SLA so failures surface at 04:30 with time to react, instead of at 06:00.
Pause on repeated failure. For a job that mutates bronze non-idempotently, repeated automatic retries across runs are worse than a stopped schedule.

Finally, if ingestion plus silver is genuinely ~3h40m of work, the conversation is about splitting the job or shortening the critical path, not about scheduling tricks.