8% flaky CI: integration tests asserting delivery guarantees Pub/Sub never made

"Got 4" - the tests assert exactly-once delivery, which does not exist. Pub/Sub is at-least-once. Duplicates are legal and expected: a subscriber that acks slowly, an ack that is redelivered before it is processed, or the emulator's own redelivery timing can all produce a second copy. The test is asserting a property of the product, not of your code, so it is guaranteed to be flaky under load. The invariant that actually matters is that your handler's effect is idempotent (dedupe on order ID). Test that instead: publish a duplicate deliberately and assert one business effect, not one message.

"Got 0" - a lifecycle race and cross-test contamination. Two mechanisms fit:

• Subscription created after publish. A subscription only receives messages published after it exists. If the test publishes and then creates the subscription (or does both concurrently), the messages are gone. Ordering must be create-subscription → publish → consume.
• Shared topic, sequential tests. All tests use orders-test. A slow goroutine or leftover message from the previous test can be consumed by the next one, or consumed by the wrong handler, giving both false positives and false negatives. The emulator keeps state for the life of the step, so "sequential" does not mean "isolated".

time.Sleep(2s) is a race with a fixed budget. Under CI load, 2 s may not be enough (got 0); when things are fast, it may catch a duplicate (got 4). Neither outcome is a statement about correctness.

Split by what each test is really checking.

• Unit layer (should be most tests): call the handler directly with an in-memory publisher/subscriber interface. Deterministic, milliseconds, no emulator. This is where the idempotency test lives - feed the same order ID twice, assert one write.
• Contract layer (2–3 tests): against the emulator, verifying serialization, attribute mapping, and ack/nack behaviour. Keep this surface small; it is the part that can be flaky.

Make emulator tests hermetic.

• Unique resource names per test: orders-test-<testname>-<rand> for both topic and subscription, with t.Cleanup to delete them. The emulator supports arbitrary project IDs and names; isolation is cheap.
• Strict ordering inside each test: create subscription → publish → consume. Never rely on publish-before-subscribe.
• Replace Sleep with an eventually loop: poll for the expected condition with a 10 s deadline and a 50 ms interval, failing with a diagnostic dump (received messages, timestamps) rather than a bare count mismatch.
• Assert on effects (deduplicated records in the fake store), not raw message counts, and where counts are unavoidable assert >= expected plus idempotent effect.
• Consume into a buffered channel with a timeout context so a stuck receive fails fast instead of hanging the step.

Parallel-safety. With per-test resources, tests can call t.Parallel() and the package can be run with go test -race -count=1 -parallel 8. Do not share a package-level emulator client with mutable state; construct per-test clients.

Assert the guarantee you actually depend on. Write an explicit test named for it - TestHandler_DuplicateDelivery_ProducesOneOrder - so the at-least-once assumption is documented in code and nobody re-introduces an exactly-once assertion.

Diagnose first. go mod download in every step means the module cache is being rebuilt repeatedly: Cloud Build steps share /workspace but not $GOPATH/$GOCACHE unless you point them at a shared path. Running go test ./... in several steps recompiles the same packages each time. That alone plausibly accounts for most of the 15 minutes.

Fixes, in order of payoff:

1. One shared toolchain state. In every Go step, export GOPATH=/workspace/go, GOCACHE=/workspace/.gocache, GOMODCACHE=/workspace/go/pkg/mod, and stage the steps so module download happens exactly once. For cross-build persistence, cache those directories to a GCS bucket keyed by go.sum hash and restore at the start.
2. Test once, not five times. A single step running go test -race -count=1 -coverprofile=... ./... replaces the repeated runs; publish the coverage file as an artifact.
3. Parallelize independent work in cloudbuild.yaml with waitFor: lint, unit tests, vulnerability scan, and container build have no ordering dependency and should run concurrently rather than serially.
4. Layer-cache the image build (Kaniko --cache=true --cache-ttl=168h, or docker build --cache-from) so dependency layers are not rebuilt on every commit.
5. Right-size the signal per trigger. PR triggers run lint + unit + contract tests; the emulator suite and full image build run on merge to main. Use -short to gate the slow subset. This is not "reducing signal" - it is moving expensive checks to the point where they gate something.
6. Machine size. Bump the build machineType to a higher-CPU tier; Go tests parallelize well and the wall-clock saving is often larger than the cost delta.

Measure before and after per step; if a step is still slow after caching, look for tests that actually sleep - they usually trace back to the same missing isolation as the flakes.