A stream that dies after VACUUM, with a bad deploy waiting to be rolled back

Diagnosis The error means the Delta transaction log references data files that no longer exist. VACUUM ... RETAIN 168 HOURS keeps seven days; the stream has been running for eleven and pins a snapshot/log version older than that horizon, so VACUUM deleted files that version still needs. Files removed by VACUUM are gone - RESTORE cannot bring them back, because the log entry survives but the underlying objects do not.

That is also why restarting from the same checkpoint fails immediately: the checkpoint resumes at an offset whose source version is already broken, so it re-fails at the same point.

Immediate recovery

1. Do not disable the retention safety check to vacuum harder, and do not raise retention now - that only prevents the next occurrence.
2. Start the stream with a new checkpoint location and accept reprocessing of the affected window. Replay is only safe if the sink is idempotent: ensure the write uses a MERGE on a stable key, or Delta idempotent writes with txnAppId/txnVersion per batch.
3. Reconcile the output for the reprocessed window against bronze to confirm no duplicates were introduced.
4. Restart downstream consumers so they pick up the corrected data.

Separate the two risks

• The batch job reads a single current snapshot and finishes; VACUUM does not delete files belonging to the current version, so a 7-day retention is not inherently unsafe for batch. What breaks is time travel beyond retention, and long-running batch reads that span a vacuum on files they read - check the batch runtime against the maintenance window.
• The stream is the constraint. Retention must exceed the maximum age of any live reader snapshot, not just the longest job runtime. If streams are restarted only when they fail, that age is unbounded.

Policy Set retention above the longest expected reader lifetime (for example 30 days for high-value silver tables), and enforce operational discipline: restart long-running streams on a fixed cadence (weekly) so no reader outlives the retention horizon. Alert when a stream's uptime approaches the retention window. Keep 7 days only for tables with no streaming readers and no time-travel requirement - the extra storage from longer retention is far cheaper than the outage just experienced.

Also confirm the checkpoint and the table live in storage governed by the same lifecycle policies; an object-lifecycle rule deleting old files has the same effect as an over-aggressive VACUUM and is easy to miss.

No - fix first, then decide whether the deployment is implicated at all. The error is a data-availability problem caused by VACUUM, not by the trigger interval or worker count; rolling back would not make the missing files reappear and would add churn to an incident that is already confused.

That said, the deployment should be handled properly once the pipe is flowing:

• Bundles are declarative, so rollback means redeploying the previous git commit or tag to the same target, then databricks bundle run to restart the job. Hand-editing the workspace creates drift between the bundle and the deployed state - don't.
• Changing trigger interval and worker count does not break streaming semantics: the stream can restart with the same checkpoint, provided the query logic itself is unchanged. Changing the query (adding an aggregation, changing the source) requires a new checkpoint.
• Use repair run for the failed task only; do not rerun already-succeeded tasks that are not idempotent.
• Add the rollback path to the runbook and rehearse it in a staging target, because 'redeploy the previous commit' is only a fast answer if someone has done it before.