Promotion Governance in Unity Catalog and CI/CD
A data platform supports 20 data scientists and six production models. Promotion is manual: a scientist registers a model version in the workspace registry, edits a notebook to point the production job at the new version, and messages the platform team. Last week version 15 was registered without a model signature; the serving endpoint rejected it after the job had already switched. Another scientist moved the 'Production' stage on version 14 to a different model, so the rollback target was unclear. Audit cannot identify who approved version 15, which Git commit produced it, or which training data was used. The platform team wants a repeatable promotion process with least privilege, immutable versions, environment separation, and fast rollback, using Unity Catalog and Databricks Asset Bundles.
1. Design the registry and permission model. Explain how to prevent signature-less versions and ambiguous promotion targets.
Read the worked answer
Use the Unity Catalog model registry and treat every model version as immutable. Training jobs register a new version; no one edits or overwrites an existing version. Require a model signature and input example at registration time, enforced by a CI check and a serving smoke test. Attach tags or metadata for git_sha, training table version, MLflow run ID, evaluation metrics, and approver. Replace mutable stages with aliases such as Champion and Challenger; alias changes are explicit and auditable. Grant scientists read and execute permissions on training jobs, but restrict alias changes and endpoint updates to a service principal or an approver group. Separate dev, staging, and prod catalogs or schemas. Promotion should copy a reference, not mutate production in place. Record approval in the job run or pull request. This prevents both the missing-signature deployment and the ambiguous rollback target.
2. Describe the CI/CD flow with Databricks Asset Bundles from commit to production alias. Include the checks that must fail the pipeline.
Read the worked answer
Define jobs, notebooks, and endpoint configuration in an Asset Bundle stored in Git. The pipeline should validate data schema and feature order, train in a dev or staging environment, evaluate against a baseline, and register a version with tags. Run a staging endpoint test using a representative payload and assert the signature, response schema, p99 latency, and score distribution. Require an approval gate before production. Then deploy the bundle and update the Champion alias or traffic routing through a service principal, not a personal token. The pipeline must fail if the signature is missing, feature order differs, evaluation is below threshold, or the integration test times out. Store secrets in secret scopes. Version endpoint configuration alongside the job. Rollback is then a Git revert plus an alias or traffic change, not an emergency notebook edit.
3. A bad version reaches production. Explain the rollback and audit procedure using immutable versions and aliases.
Read the worked answer
Because versions are immutable, rollback does not require retraining. Keep the previous Champion version and endpoint configuration. Atomically move the Champion alias back to the prior version, or shift 100% of endpoint traffic to the prior served entity. Verify with a smoke test and compare inference-table latency and score distribution. Audit the incident using registry tags, MLflow run ID, job run ID, git_sha, training table version, approver, and alias-change audit logs. The inference table provides request and response evidence. Do not delete the bad version; tag it as rejected or quarantined. Add a regression test reproducing the failure before allowing the next promotion. Test rollback in staging quarterly so it is not discovered during an incident.