Databricks Data Engineer: Implementing CI/CD

Bundles are the unit of deployment

Databricks Asset Bundles (DABs) describe jobs, pipelines, notebooks, and clusters as code in databricks.yml. The file contains a bundle name, variables, resources (jobs, pipelines, and so on), and one or more targets (dev, staging, prod), each specifying a workspace host and overrides. Commands:

• databricks bundle validate - schema and reference checks, changes nothing.
• databricks bundle deploy -t prod - uploads files and creates or updates resources in the target workspace, storing state under .databricks/bundle/.
• databricks bundle run <resource> -t prod - executes the deployed resource (deploying first if needed).
• databricks bundle destroy - removes deployed resources.

Override variables with --var key=value or per-target variables blocks. Resource paths are relative to the bundle root, so notebook_path: ./src/ingest.py resolves inside the bundle.

CI/CD flow

Commit → CI runs bundle validate plus unit tests → deploy to the dev target → integration tests → promote the same commit by deploying to the prod target. Authenticate CI with a service principal using OAuth machine-to-machine credentials (or a PAT held in CI secrets); never commit tokens. Use Git folders for interactive development and separate workspaces per environment so production is unreachable from development.

Traps

• Thinking deploy runs the job - it only deploys; run executes.
• Hard-coding hosts and catalog names instead of variables, which breaks promotion.
• Using a personal user's token in CI, so the job owner becomes a person who may leave the team.
• Sharing one workspace across dev and prod with no target isolation.