Ordering keys that don't order, a 600-second lease, and a dead-letter topic that isn't dead

What the guarantee actually is. Pub/Sub ordering is per ordering key, and it constrains delivery order to the subscriber. It says nothing about the order in which your code finishes processing them. With a push subscription and concurrency=80, several messages sharing an ordering key can be in flight simultaneously - on the same instance or on different instances - so the Firestore write that lands last is whichever request finishes last, not whichever was published last. Nacks and redeliveries reorder further: a redelivered message interleaves with fresh ones.

Options, most trusted first:

1. Make order irrelevant (recommended). Carry a monotonic version (or sequence number) per SKU in the payload. Write inside a Firestore transaction that reads the stored version and rejects the write if incoming.version <= stored.version. This is correct under arbitrary reordering, duplicates, and concurrent instances, and it costs one extra field read per write. Ordering becomes an optimization rather than a correctness dependency.
2. Pull subscription with an ordering-aware client library. The Go/Java clients support ordering keys and invoke the callback serially per key when ordering is enabled. This preserves order, but forces you to move off push, tune MaxOutstandingMessages, and accept that a slow key head-of-lines everything behind it.
3. Serial processing per key with a distributed lock (e.g., Firestore/Redis lease per SKU) around a concurrency=80 service. Works, but it is a hand-built scheduler with its own failure modes (lock expiry during GC pauses) and it silently converts your throughput ceiling into "one in-flight per SKU".
4. Set Cloud Run concurrency=1. Correct but expensive: you cap the service at one request per instance and pay for instances proportional to throughput.

Option 1 is the only one that survives every failure mode in this list, including the redelivery problem in the next question - so it is the answer even if you also adopt 2.

The mechanism is arithmetic. The maximum ack deadline is 600 s. The handler takes 720–900 s. It is structurally impossible for this handler to ack in time, so Pub/Sub redelivers at 600 s - while the first attempt is still running. Now two copies run concurrently, both eventually write, and the next redelivery stacks a third. The duplication is not a bug in Pub/Sub; it is the contract being violated.

Critically: a push endpoint cannot extend the ack deadline. There is no ModifyAckDeadline call available to a webhook, so no amount of retry-policy tuning fixes this. Options:

1. Ack fast, work asynchronously (best fit for nightly batch reconciliation). The push handler validates and enqueues the work - write a job record, publish to a work topic, or trigger a Cloud Run Job / Workflows execution - then returns 200 in well under a second. Long-running work no longer lives inside an ack window. You must carry an idempotency key (a run ID per day/SKU) so a duplicate trigger is a no-op.
2. Switch to a pull subscription with streaming pull. The client libraries extend the lease automatically in the background, so a 15-minute handler is fine. Cost: you now operate a consumer (GKE/Cloud Run worker, or a Dataflow job) instead of an HTTPS endpoint.
3. Shard the work so each message is under ~8 minutes, leaving margin for the 600 s ceiling. Good if the work is genuinely partitionable and each shard is independently idempotent.

Related tuning regardless of choice: raise max ack deadline to 600 s explicitly (it is not the default), and reduce batch size per message so a single message maps to a bounded unit of work. Do not rely on "the handler usually finishes in time" - the tail is what gets redelivered.

No - the policy is measuring the wrong thing. With min backoff 10 s, five attempts elapse in roughly a minute (10 + 20 + 40 + 80 + 160 s of backoff, plus processing). That means any transient blip of ~1–2 minutes permanently diverts messages to the dead-letter topic. Meanwhile the messages that should be quarantined - malformed payloads that will never succeed - are indistinguishable from the slow ones. You have a policy that dead-letters healthy traffic and gives you no signal about genuinely poisonous messages.

Changes:

• Set min backoff to ~60 s and max to 600 s, and raise max delivery attempts to something like 20–30. A slow-but-eventually-successful handler then recovers on its own. Pair this with the async pattern from the previous question so retries are not also duplicating work.
• Treat the DLQ as "poison", not "slow". If a message fails 30 times over several hours with a validation error, it belongs there.
• The DLQ must have its own subscription - otherwise the forwarded messages are discarded. It also needs the Pub/Sub service agent granted publish on the DLQ topic and subscribe on the source subscription, or forwarding silently fails.

Monitor:

• subscription/dead_letter_message_count - alert on any sustained non-zero rate; a healthy system is zero.
• subscription/num_undelivered_messages and subscription/oldest_unacked_message_age - the leading indicator that consumers are falling behind (not just the lagging DLQ counter).
• Ack latency distribution and Cloud Run 5xx rate, to separate "consumer broken" from "consumer slow".

Operate: build a redrive path before you need it - a small job that reads from the DLQ subscription and republishes to the source topic with the original ordering key preserved, gated on the consumer being healthy. Without it, the DLQ is a graveyard people are afraid to touch.