Hand-Deployed Jobs Across Three Workspaces

Adopt Databricks Asset Bundles with one bundle per project and one databricks.yml containing dev, staging, and prod targets that differ only in workspace.host and variables (catalog, warehouse id, cluster policy).

Everything that defines prod goes in the bundle: job definitions with their schedules, notification destinations, parameters, and permissions; pipelines; and the notebooks or wheel artifacts they reference. Those are exactly the two things that were lost in the incident, and both are declarative fields, so they get reviewed in a pull request instead of re-created through a UI.

Use mode: development for the dev target so resources are prefixed per user and schedules are paused, and production mode for staging and prod so names and schedules are literal. CI runs databricks bundle validate on every pull request and databricks bundle deploy --target prod only on merge to main. Deployment happens from the CI runner, never from a laptop, so the deployed artifact is tied to a commit.

Replace the personal access token with the service principal using OAuth machine-to-machine: a client id and client secret issued for the SP, with the secret stored in the CI system's secret store and injected at run time. Personal tokens tie deployments to an individual's workspace membership and disappear when they leave.

Ownership is the other half. Set run_as on the deployed jobs to the service principal so schedules, run history, and any queries execute as a stable identity rather than as whoever last clicked deploy. Grant the SP the Unity Catalog privileges it needs (catalog/schema usage, table select and modify) through group membership so the grants are auditable in one place.

Permissions that consumers need belong in the bundle's permissions block against account groups, not individual users. Secret scopes are not a bundle resource, so provision scopes and secrets via a bootstrap job or Terraform and reference them in bundle config with {{secrets/scope/key}}, so no secret value ever appears in Git or in a notebook.

The incident is a drift problem, not a tooling problem, so the fix has three layers:

1. One writer. Prod is written by CI only. Humans get read access to the prod workspace, and the team convention (enforced by policy where possible) is that nobody edits prod resources in the UI. A UI edit will be reverted by the next deploy, which is the desired behavior, but it should be a rare event.
2. Reviewable diffs. The pull request shows the databricks.yml change, so a deleted schedule or a removed notification destination is visible as a diff line. Branch protection plus an environment approval gate on the prod target makes that review mandatory.
3. Deployment provenance. Tag the deploy with the commit SHA and record it in the release notes or job parameters; then 'which commit is live' is answerable without archaeology.

For detection, run a scheduled CI job that validates and plans the prod target against the live workspace and reports differences. That catches UI drift and unauthorized CLI deploys within a day instead of at the next incident.