Lineage and network isolation for a 4 TB/day telemetry pipeline

The mistake to avoid is 'registering a dataset' by copying or exporting data into the workspace's default storage - at 4 TB/day that is both unaffordable and creates a second, drifting copy.

• Datastore, not keys. Register ADLS Gen2 as a datastore using credential-less (managed identity) authentication. Jobs authenticate as the compute's managed identity, so no secret ever reaches a notebook or an environment variable.
• Data assets with immutable versions. Register the telemetry as a data asset (URI type uri_folder pointing at the ADLS path) and create a new version for each logical snapshot - daily or release-based. A version records the path and the time it was captured, which is what satisfies the 18-month retrain requirement without duplicating bytes.
• Reference versions, never raw paths. The job must consume azureml:telemetry:3, not abfss://.... The version pins the input.
• Log the rest. Log metrics, parameters, and the model with MLflow so the registered model carries a link back to the run, and the run carries the input asset versions.

The auditor's question then resolves to: registered model -> run ID -> logged input asset versions -> ADLS paths and capture timestamps.

Use a workspace managed virtual network with 'allow only approved outbound' isolation, plus private endpoints from that VNet to ADLS Gen2, Key Vault, Container Registry, and the workspace's dependent resources. Put compute instances (authoring) and compute clusters (jobs) in the same managed VNet so both paths are covered - a private endpoint on storage alone does not help if a compute instance outside the VNet tries to reach it.

What breaks first is package installation. With only-approved-outbound, jobs can no longer pull from public PyPI or Docker Hub at runtime. The fix is upstream of the job: curate environments in a private ACR, publish approved packages to an internal index reachable over a private endpoint, and pin environments by name and version rather than letting the script pip install at start. Expect the first week of failures to be image pulls and pip timeouts, not data access.

Second thing to break: interactive notebook conveniences - Git integration, VS Code extensions, and telemetry to public endpoints. Decide explicitly which of these you are willing to lose, because 'allow internet outbound' would silently undo the security requirement.

The naive answer - tag jobs with a team name and read the cost report - fails because Azure Machine Learning compute bills at the cluster level (node-hours), not per job, and cost management has no view of individual runs.

The practical options, in order of effort:

1. Separate clusters per team, autoscaling with min nodes = 0, each tagged with the team. Cost allocation becomes exact and trivial, and idle clusters cost nothing beyond the OS disk. This is usually the right call with only three teams.
2. One cluster, derived attribution. Keep the shared cluster but have every job emit a team tag and log vCPU-seconds / node-hours; then reconcile total node-hours from the cluster's billing against the derived per-job split. Cheaper in quota, but the numbers will not tie exactly and someone must own the residual.

Do not promise per-job billing accuracy on a shared cluster. If the cluster must be shared - because of a constrained vCPU quota - say so up front and agree that attribution is an estimate. The 6-18 hour run times also argue for dedicated clusters: a shared cluster means one team's long job blocks another's.