I85PREP
⌘ K
Azure Databricks Data Engineer Associate study materials
Azure Databricks Data Engineer Associate study resources

Azure Databricks Data Engineer Associate: Implementing CI/CD

Azure Databricks Data Engineer Associate · Work through your exam, one topic at a time.

All guide sections

Implementing CI/CD

Implementing CI/CD

This domain tests promoting Databricks code and configuration through environments with source control, automation, and identity. Questions ask how to deploy jobs and pipelines repeatably, authenticate, and test.

CI/CD rules

  • Git folders (formerly Repos) connect a workspace to Git for development, but deployment should be code-driven. Databricks Asset Bundles define jobs, pipelines, and workspace files in databricks.yml, with targets for dev/test/prod and variables for environment-specific values. Commands include bundle validate, bundle deploy, bundle run, and bundle destroy.
  • Terraform is appropriate for account/workspace infrastructure and UC objects; bundles are appropriate for data and job assets. Do not mix ownership of the same object between tools.
  • Authentication: use a Microsoft Entra ID service principal with OAuth machine-to-machine (M2M) or a federated credential in CI. Avoid personal access tokens in pipelines. Store secrets in Azure Key Vault-backed secret scopes or the CI secret store.
  • Pipeline stages: lint and unit test, validate the bundle, deploy to dev, run integration tests, then deploy to prod with approval. Build once and promote the same artifact; parameterize paths, catalogs, and warehouse IDs per target.
  • Testing: pytest with local Spark or Databricks Connect for transformation units; integration tests against a dev workspace; test notebooks as jobs, not manually.

Worked example

A team wants a job deployed to dev on every commit and to prod after approval. Reasoning: commit databricks.yml with dev and prod targets. Define a variable for the catalog and storage path. The CI service principal authenticates with OAuth M2M; the pipeline runs bundle validate, deploys to dev, runs the job, then waits for approval to deploy prod. Prod uses its own service principal and workspace, so no dev credentials reach production.

Traps

  • Committing secrets or storage keys to Git.
  • Using a PAT for CI when service-principal OAuth is available.
  • Deploying manually from a Git folder and calling that CI/CD.
  • Using the same workspace and catalog for dev and prod.
  • Hardcoding environment values instead of bundle variables.
  • Treating bundle deploy as also running tests.