Autoscaling stalls at 40 workers while the backlog grows

The signature - lag rising, CPU low, worker count flat - means the work is not spreadable, not that the job needs more VMs.

1. Hot key. 30% of 25,000 msg/s is 7,500 msg/s under one account_id. Every bundle for that key is processed by a single worker; the other 39 have little to do, which is exactly the 22% average CPU. --maxNumWorkers=200 cannot help because parallelism is bounded by the key space, not by the quota. Fix: two-stage aggregation - salt the key (account_id plus a bucket from hash(event_id) % 20), partial-aggregate, re-key on account_id, then finalize - or Combine.perKey(...).withHotKeyFanout(20). Fanout trades state and shuffle for parallelism.

2. Interrogate the enrichment. A per-element BigQuery call is one RPC per element: 25,000/s is 2.16 billion lookups a day, and on-demand billing has a 10 MB minimum per query. Even if it is not the lag source, it is a prime suspect for the doubled bill. Replace it with a broadcast side input (refresh every 10-15 min) or a low-latency KV store.

3. Look for a non-throughput scaling blocker. Dataflow also refuses to scale on memory or disk pressure - check worker logs for OOM and harness restarts - and on regional quota: a project capped at 40 workers' worth of vCPUs in that region looks identical to this.

4. Bound state. Unbounded per-key state inflates checkpoint time and can pin the pipeline regardless of worker count; set windows or state TTL.

Only after 1-3 is 'more workers' a legitimate lever - and then the right move is raising --minNumWorkers to cover the 25,000/s baseline so you are not cold-scaling on every burst.

Alert on user-visible symptoms, not on worker counts.

• Primary page: Pub/Sub oldest_unacked_message_age above 300 s for 5 minutes. That is the freshness SLO, and unlike system_lag it is not a per-bundle high-watermark that jumps around.
• Secondary: multi-condition policy on Dataflow system_lag combined with is_failed, and a rate-of-change alert on num_undelivered_messages - a growing backlog is the leading indicator.
• Also alert on dataflow.googleapis.com/job/is_failed and on sink watermark staleness.
• Use fast-burn (5 min) and slow-burn (1 h) thresholds so a transient spike does not wake anyone.
• Explicitly do not alert on average CPU. This incident proves that low CPU is the symptom.

Deploying the fix. Changing the aggregation key breaks state compatibility, so an in-place --update will be rejected or will produce garbage accumulations:

1. gcloud pubsub snapshots create on the subscription, then stop the old job with --drain (or cancel once the snapshot exists) and start the new job with --pubsubSnapshotId. No messages lost; the sink's exactly-once write absorbs any overlap.
2. Canary first: a second 1% subscription feeding a staging copy of the pipeline, compared against production numbers for an hour before the swap.
3. Rehearse correctness on the batch path - the same transforms over a bounded source validate the new aggregation without touching live traffic.
4. Keep the previous job spec and job ID in Terraform for rollback; replaying from the GCS raw archive is the last resort.
5. Deploy from a versioned template, not a hand-typed gcloud dataflow jobs run. The on-call engineer needs one command at 03:00.

Split the workloads. They share transforms, not resources. Package the transforms as a library consumed by two pipelines. Running batch bursts and steady streaming in one job forces you to pay for streaming-sized capacity all day.

Batch backfill (4 TB/night). At ~$6.25/TB the BigQuery scan is only ~$25, so the bill there is VM-hours. Use FlexRS (--flexRSGoal=COST_OPTIMIZED), which takes Spot capacity with a 6-hour SLA that a nightly job can absorb. Set a hard --maxNumWorkers and pick a machine shape matched to the transform (I/O-bound versus CPU-bound), not the default.

Streaming. Keep Streaming Engine - it moves state and shuffle off worker disks, so you can run smaller machines and fewer of them, and it is what makes the hot-key fix affordable. After the topology fix, set --minNumWorkers to the level that handles 25,000 msg/s and let autoscaling cover peaks. A minNumWorkers frozen at the old stuck value of 40 simply pays for idle workers every month.

The hidden line item is the enrichment. Per-element BigQuery queries at 25,000/s, with a 10 MB minimum billed per query, are the most plausible driver of the doubled bill. Replacing them with a broadcast side input or a KV lookup removes essentially all of it.

Prove the savings. Compare vCPU-hour and shuffle metrics before and after, and label both pipelines so a billing export can separate streaming from batch. Without the label split, the next cost review cannot tell which change worked - and the workload you cannot attribute is the one you cannot control.