A 25-minute rollback: pinned traffic, deleted revisions, and migrations on startup

Three compounding decisions:

1. Traffic pinned to "latest". Every successful deploy immediately takes 100% of traffic. There is no phase in which a bad build is visible to a small number of users or no users.
2. Revisions deleted by a cleanup job. Revisions are immutable, cheap, and are the rollback unit. Keeping three is arbitrary and, because deploys are frequent across 30 services, three revisions can be hours old - or already gone before the incident is even understood.
3. Rollback therefore became a rebuild. With no addressable previous revision, recovery time is dominated by cloning an old commit, rebuilding, and re-deploying. That is the 25 minutes.

Highest-leverage single change: stop deleting revisions, and deploy with --no-traffic so the previous revision stays addressable.

Concretely: keep revisions for N days (or keep anything carrying a latest-good tag) instead of "three newest", and change the deploy to publish a new revision with no traffic first. Rollback then becomes:

gcloud run services update-traffic orders-api --to-revisions=orders-api-00417=100

which is a traffic-table update - seconds, not minutes, and it does not depend on the registry, the build, or the source repo. Note that if the bad release did write to the database, traffic shift alone is not sufficient; see the schema-change strategy in the next question.

Release sequence (per service):

1. Build and push the image; record the digest.
2. gcloud run deploy --no-traffic --tag=cand-<short-sha> - the new revision serves only via its tag URL. Tag URLs route directly to that revision, so it is testable before any user traffic arrives (mind auth: tag URLs still enforce the service's auth policy).
3. Gate: run smoke tests against the tag URL - a health endpoint, one read path, one write path against a test tenant.
4. Gradual shift with automated verification: 1% → 10% → 50% → 100%, with a wait and a check at each step on error rate and p95 latency from Cloud Monitoring. Abort on breach by shifting traffic back to the previous revision. Cloud Deploy models exactly this as a canary strategy with a verification job; use it rather than hand-rolling the loop.
5. On success, tag the revision latest-good and only then allow traffic to advance further. Never pin to latest.

Schema changes - expand/contract, never on container start:

• Run migrations in a Cloud Run Job (or Cloud Build step) that completes before the new revision receives traffic, not in the container entrypoint. This removes the concurrent-startup deadlock entirely and gives one failure surface.
• Only additive, backward-compatible DDL in the same release as code: add nullable columns, add tables, add indexes concurrently. Old and new revisions must both work against the schema during the traffic shift.
• Drop or rename columns in a later release, after 100% of traffic and all old revisions are on code that no longer reads them.
• Keep an advisory lock in the migration job for safety, and have the application fail fast at startup if the schema version is unexpected rather than attempting to migrate.

Why this matters for rollback: with expand/contract, the previous revision remains valid against the new schema, so a traffic shift is a complete rollback, not a partial one.

Credentials.

• Move the database credentials to Secret Manager and reference them at deploy time: --set-secrets=DB_PASSWORD=orders-db-password:latest (or pin a specific version for reproducibility rather than latest).
• Grant the runtime service account roles/secretmanager.secretAccessor on that one secret, not project-wide.
• Rotate the exposed credential immediately and purge it from the repo. It is in git history, so treat it as public: rotation precedes cleanup. Add a secret scanner (e.g., Cloud Build + a pre-commit hook) so it cannot recur. Note that --set-env-vars also exposes the value in revision metadata and console output, which is a second reason the secret must never travel that path.

Blast radius.

• roles/run.admin at project level means one compromised build script can redeploy, or delete, all 30 services. Split identities:
  • A build SA: write to Artifact Registry, read source, no deploy rights.
  • A deploy SA: roles/run.deploy scoped per service via IAM conditions on resource name, or at minimum roles/run.developer plus roles/iam.serviceAccountUser only on the specific runtime SAs.
• Pin --service-account explicitly on each service so deploys cannot silently run as the default compute SA.
• Enable Binary Authorization in require-attestation mode, with an attestation produced by the pipeline. This makes "only images built by this pipeline can run" an enforced property rather than a convention.
• Put the trigger on a private pool with a scoped VPC/egress policy, and require PR review on cloudbuild.yaml itself via CODEOWNERS - the pipeline definition is production code.