Consolidating a Fragmented Azure ML Estate Without Losing Lineage
A financial-services company runs 45 data scientists across 6 teams on Azure Machine Learning. Each team owns its own workspace, model registry, and AKS inference cluster; the same feature-engineering code lives in four repositories. Production scoring for 11 models runs on two clusters sized for peak load, averaging 18% GPU utilization. Security mandates private endpoints everywhere with no public egress. The platform team owns Terraform, but teams edit compute quotas directly in the portal, which caused three quota-exhaustion failures last month. Audit requires per-model lineage from training data snapshot to deployed endpoint, plus rollback within 15 minutes. Leadership wants fewer workspaces; teams resist sharing a registry because of recurring model-name collisions.
1. Recommend a workspace and registry topology. Justify it against the audit requirement and the teams' autonomy concerns.
Read the worked answer
Decision: two workspaces (prod isolated, non-prod shared) plus one central registry per tenant.
Reasoning:
- Lineage is anchored in the registry. Six separate registries is precisely what makes audit expensive today, so consolidation must start there. Naming collisions are a governance bug, not an isolation requirement: enforce
{team}-{model}-{stage}via CI policy on registration and reject non-conforming names. Nothing about a shared registry forces shared ownership. - Separate prod from non-prod because the failure domains differ: prod carries RBAC for the scoring service and its own quota; dev/test churn would otherwise consume prod capacity.
- Sharing one non-prod workspace across 6 teams only works with per-team compute quota and per-team resource groups; without that, you have recreated the quota problem inside one workspace.
Rejected: 6 workspaces (lineage fragmentation, 6x private-endpoint and DNS cost), 1 workspace (prod blast radius, audit noise).
Test of the design: can you answer 'which data snapshot and code commit produced the prod endpoint for model X' in under five minutes from registry metadata alone? If not, the topology is not done.
2. Design the promotion path from dev to prod given private networking and Terraform ownership.
Read the worked answer
Separate artifact promotion from environment provisioning.
- Teams push code and pipelines; the pipeline registers a model version in the shared registry with immutable tags (git SHA, data snapshot ID, eval report URI).
- Promotion is a registry operation: dev -> test -> prod changes the stage/alias on a version that already passed gates. No rebuild between environments; rebuilding is where drift between what was evaluated and what was deployed creeps in.
- Environments are Terraform-only. Private endpoints, DNS zones, and compute definitions are declared in IaC. Teams get no portal write access in prod; they request quota by pull request, which makes quota a reviewed change and kills last month's failure mode.
- The 15-minute rollback comes from pointing the prod alias at the previously approved version behind a stable endpoint name, not from rebuilding a container.
Practical constraint to surface: if the artifact cannot reach the registry from a no-egress build agent, the promotion chain breaks silently. Verify private DNS resolution from the build agent to the registry before promising the flow.
3. The platform team proposes freezing all model deployments for two weeks to migrate to the new topology. Evaluate that proposal.
Read the worked answer
Treat the freeze as a cost/risk trade, not a correctness question.
- Estimate what two weeks of frozen deploys costs: delayed model fixes, drift exposure on the 11 live models, and the pressure that builds to bypass the freeze at week two (which is how portal-edited quotas happened in the first place).
- The migration does not require a global freeze. It requires a cutover order: migrate non-prod workspaces first, prove lineage and rollback on a low-risk model, then move prod endpoints one model at a time behind stable endpoint names.
- Reserve a hard freeze only for the specific window where DNS or identity changes affect already-deployed endpoints, measured in hours, not weeks.
Offer a compromise: freeze new prod models during the prod cutover window, allow bug-fix redeploys of existing approved versions, and require the expedited path to go through the same alias mechanism so it is still auditable. If the platform team cannot state which migration step actually breaks live scoring, the freeze is precaution rather than engineering.