From an 8,000-row notebook to 300M transactions with a 50 ms serving constraint
A fraud team prototyped a gradient-boosted model in a Vertex AI Workbench notebook. It reads 8,000 rows from a CSV with pandas, engineers 40 features using pandas groupby over per-card history, trains XGBoost in about 90 seconds, and reaches AUC 0.91 on a random split. The production dataset is 300M transactions (~45 GB) in BigQuery with 18 months of history; features need trailing 90-day windows per card. The team wants weekly retraining, and the same feature logic must run at prediction time for a single transaction in under 50 ms. There are two data scientists, one of whom insists on keeping the notebook workflow.
1. The AUC of 0.91 will not survive contact with production. Identify the two most consequential flaws in the prototype before scaling it, and explain what each would do to the model.
Read the worked answer
Random split on temporal fraud data. Fraud patterns move; a random split puts future transactions in the training set and near-duplicate card histories on both sides. The 0.91 is inflated and will drop at deployment. Split by time: train on months 1-12, validate on 13-15, test on 16-18, and report the test slice.
Per-card groupby features computed over the same rows being predicted. If trailing aggregates are computed over the full dataset before splitting, each row's features are contaminated by later rows from the same card. The fix is the same as the production design: windows must be strictly backward-looking (ROWS BETWEEN 90 PRECEDING AND 1 PRECEDING), computed against data available at scoring time.
Both flaws inflate offline metrics in the same direction, so fixing one and not the other will still look like a regression. Fix both, take the lower honest number, and use it as the baseline that the production model must beat - otherwise the team will spend a quarter chasing an unreachable offline score.
2. How do you move the 40 pandas features into a pipeline that trains weekly on 45 GB and serves single transactions in under 50 ms, without rewriting them twice?
Read the worked answer
One definition, materialized twice.
Training: implement features as BigQuery SQL with window functions partitioned by card_id and ordered by event time, materialized into a partitioned training table, then exported as sharded Parquet or TFRecords to GCS. Train with a Vertex AI custom training job running distributed XGBoost, reading shards directly - never pull 45 GB into a pandas frame. If hyperparameter tuning needs are modest, BigQuery ML boosted trees is a legitimate shortcut that removes the export entirely.
Serving: you cannot compute a 90-day window in 50 ms per request. Precompute card-level trailing aggregates on a schedule and at event time (Dataflow streaming into Bigtable, or Vertex AI Feature Store online serving with batch ingestion from the same SQL), then the ranker fetches 40 precomputed values by key and scores. Target under 10 ms for retrieval, leaving budget for the model.
Keeping them equal: the SQL is the single source of truth in a repository; the batch job and the streaming/batch feature-load job consume the same definition. Add a daily reconciliation job that computes features both ways for a sample of cards and alerts on divergence - training-serving skew is the failure this design exists to prevent.
The notebook stays as the place to explore. Promotion goes through a Vertex AI Pipeline with components, caching, Vertex ML Metadata lineage, and Cloud Scheduler for the weekly trigger. That satisfies the scientist who wants the notebook without letting it be the production artifact.
3. The team wants to retrain weekly and deploy automatically if the new model looks better. What gates would you put between training and production, and why is 'looks better' insufficient?
Read the worked answer
'Better' on aggregate validation AUC is insufficient for a fraud model because the cost surfaces are asymmetric and the data moves. Gates, in order:
- Data validation - row counts, null rates, schema, and per-feature distribution bounds versus the previous run. A silent upstream change should stop the pipeline, not produce a subtly wrong model.
- Evaluation on a time-sliced holdout with explicit slices (new cards, high-value transactions, each geography) and a minimum delta, not just 'greater than'.
- Decision-threshold check - at the operating threshold, what happens to false-positive rate and alert volume? A model with equal AUC can double the analyst queue.
- Shadow deployment - score production traffic without acting on it for a defined window, compare score distributions and, once labels arrive, realized precision.
- Human approval for promotion while the team is new to the pipeline; automate only after several clean cycles.
Automate retraining and evaluation; keep promotion behind a gate until the false-positive and drift behavior is well understood.