GCP ML Engineer (Google Cloud Certifications): Prototype classifier that must retrain weekly

Do the arithmetic before choosing. Four workers at a realistic 75-80% scaling efficiency gives roughly 19h / (4 x 0.78) ≈ 6 hours per epoch - still far outside a 12-hour window for 30 epochs. So distributed training alone does not meet the requirement, and neither does a single A100 (optimistically 3x faster = ~6.3h/epoch). The target is unreachable by hardware substitution; the recipe has to change.

Sequence it this way:

1. Fix the input pipeline first. 40% decode time is the cheapest win: pre-shard the JPEGs into TFRecord/WebDataset shards once, cache them in Cloud Storage near the training region, and stream with high parallelism. That alone can cut step time materially and costs almost nothing.
2. Then add workers. Use a multi-worker worker_pool_specs with a reduction server strategy to cut all-reduce bandwidth; a MirrorStrategy change only, no model rewrite.
3. Then change the training budget. Fine-tune from a pretrained checkpoint with a frozen backbone for most epochs, or reduce epochs with a better schedule. 30 epochs from scratch on 2.1 TB is a choice, not a constraint.

I would change course if profiling after step 1 showed decode time under 5% - then it is genuinely compute-bound and more A100 workers are the right lever.

Do not read BigQuery on every epoch. A BigQuery client inside the container re-scans the label table each epoch, which is slow, costs query bytes 30 times over, and - worse - lets the label set change mid-run if an analyst edits the table.

Instead, make a snapshot the unit of work:

1. A pipeline step materializes a dated label snapshot table (labels_YYYYMMDD) and exports it as sharded records joined to the image shard index.
2. The exported snapshot URI and the label table version are passed as pipeline parameters, so a rerun of last week's pipeline reproduces last week's model exactly.
3. The training container reads only from Cloud Storage shards - sequential, cheap, and cacheable.

For reproducibility, pin the container image by digest, not by tag, and log the snapshot URI, git commit, and hyperparameters as pipeline artifacts. When an auditor asks 'what produced the model serving traffic on 3 March', the answer is a pipeline run ID, not a notebook someone ran locally.

Also validate the snapshot before training: row count within expected range, no orphaned image references, class balance drift check. A bad label export silently poisons a weekly job.

Do not spend the budget on the full dataset. Each full-data trial costs hours, so 60 trials would never fit the window and you would tune once a quarter instead of once a week.

Structure it as:

• Tune on a 10-15% stratified subsample with early stopping and a reduced epoch count. Rank configurations by validation metric on the subsample; the ranking usually holds even though absolute numbers do not.
• Search 6-8 parameters, not 25. Learning rate, weight decay, batch size, dropout, augmentation strength, and label smoothing is enough. Vizier's advantage shrinks as dimensionality grows.
• Parallel trials of 8 to fit wall-clock, but keep the parallel count modest relative to trial duration so the service can actually use completed results to inform the next suggestions. Fully parallel random search is what you get if you set parallelism equal to total trials.
• Retrain the single best configuration on the full snapshot and record it as a normal pipeline run.

Re-tune monthly, not weekly; the weekly job should reuse the best known configuration unless validation metric degrades beyond a set band.