Two teams, one feature name, two different numbers
Two product teams share a subscription analytics platform. The marketing churn model and the product-engagement model both use a feature called active_user_30d, but marketing computes it as 'opened the app', while product computes it as 'completed a core action'. Both write to their own BigQuery tables; the numbers disagree by up to 22% for the same user on the same day. There are 14 models in production, three built by analysts who have left, and an internal audit found no record of which training data produced the current champion of the revenue-forecast model. Security wants row-level access controls on subscriber PII; data scientists want to keep joining freely. The platform team owns BigQuery; each product team owns its own Vertex AI project.
1. Design the target structure for shared features and model ownership. Be explicit about what you centralize and what you deliberately leave federated.
Read the worked answer
Centralize definitions and data, federate models and experimentation.
- One shared BigQuery dataset owned by the data platform team holds conformed features. Each team keeps its own Vertex AI project, service accounts, and quotas, so a bad training job cannot starve another team.
- Feature logic lives as versioned SQL in a repository (Dataform or dbt), reviewed like code, with distinct names for distinct semantics: active_user_30d_app_open and active_user_30d_core_action. Do not resolve the conflict by picking a winner - resolve it by making the ambiguity impossible to write accidentally. A model card must declare which version it consumed.
- Data Catalog (Dataplex) entries carry owner, definition, upstream tables, and freshness SLA. An undocumented feature is a finding, not a style issue.
- PII stays in restricted columns; data scientists query authorized views and column-level policy tags, so joins still work without exposing raw subscriber identifiers.
Publish to Vertex AI Feature Store only where online serving exists. Duplicating every batch feature into a feature store adds a synchronization failure mode for no benefit.
2. The audit finding is really a lineage problem. What has to change so that the champion revenue-forecast model can be reconstructed six months from now?
Read the worked answer
The gap is that training happened ad hoc, so nothing recorded inputs. Fix the process, not the artifact:
- All training runs through a Vertex AI Pipeline. Notebooks are authoring tools; promotion requires the pipeline. Vertex ML Metadata then captures the executed SQL, the BigQuery snapshot, hyperparameters, the container image digest, and the output artifact automatically.
- Snapshot the training data. Either a time-partitioned table with a retention policy, or an explicit export to GCS with a content hash recorded as an artifact. A query that says 'last 12 months' is not reproducible; a table with a partition range is.
- Register in Vertex AI Model Registry with a model card stating intended use, training window, feature versions, evaluation slices, and owner. Deploy by alias (champion) so the serving identity is decoupled from the version string.
- Retroactively backfill model cards for the 14 models; where the analyst has left and data cannot be reconstructed, mark the model as unlineaged and put it on a review queue with an expiry.
Expect resistance: this adds a pipeline step between a working notebook and production. Frame it as the thing that lets a model be rolled back at 2 AM without its author.
3. Security wants strict controls; data scientists want frictionless joins. Propose an access model that satisfies both, and name the tradeoff you are accepting.
Read the worked answer
Use BigQuery column-level policy tags plus authorized views in the shared dataset: raw PII columns are tagged restricted, analysts get access to views that expose hashed or aggregated identifiers, and joins across teams keep working because the join keys are available in hashed form. Add row-level access policies where a team should only see its own region or tenant.
Cross-project reads are granted at the dataset and view level, not by copying data, so there is one source of truth. Service accounts for pipelines get read-only access to views; only the platform team's ingestion identity can write raw tables. Enable CMEK and data access audit logs on the shared dataset so an unexplained export is visible.
The tradeoff: analysts lose the ability to do ad hoc joins on raw identifiers, and some exploratory work will require a request. That is the price of the audit fix - accept it, and shorten the request path to hours rather than weeks, because a slow exception process is what causes people to export CSVs to their laptops.