I85PREP
⌘ K
GCP ACE study materials
GCP ACE study resources

Cloud Run at the instance ceiling with a 1.9 GB image

GCP ACE · Put your knowledge to work in context.

All case studies

Cloud Run at the instance ceiling with a 1.9 GB image

FleetOps deploys a containerized dispatch API to Cloud Run in us-central1. The service talks to Cloud SQL for PostgreSQL over private IP and publishes to Pub/Sub. Under a 3x traffic spike last Tuesday, 12% of requests returned 503 and p99 latency reached 9 seconds; Cloud Run reported that the maximum instance count was reached. The container image is 1.9 GB and takes 14 seconds to start. The team wants to release twice a day with rollback in under a minute, and security will not allow service account JSON keys anywhere in the pipeline. Cloud Build currently builds from a GitHub trigger.

1. Diagnose the 503s and the 9-second p99, and order the fixes by expected impact. Explain why raising the maximum instance count alone is the wrong first move.

Read the worked answer

The 503s are the instance ceiling, but the ceiling was reached because each request was slow, not because traffic was unservable. Raising --max-instances alone just puts more instances in front of the same bottleneck and, worse, multiplies the pressure on Cloud SQL.

Ordered by impact:

  1. Cloud SQL connection exhaustion. A 9-second p99 against a database dependency with 5xx responses is the classic signature of requests queuing behind max_connections. Every Cloud Run instance opens its own pool, so (instances x pool size) crosses the limit during a spike. Check pg_stat_activity against the instance's connection limit. Fixes: shrink the per-instance pool, lower --concurrency so fewer connections are held per instance, and put PgBouncer in front if the instance tier cannot grow.
  2. The 1.9 GB image. A 14-second start is dominated by pulling layers, not by application boot. A multi-stage build onto a distroless or slim base typically gets this under two seconds, which directly reduces how many instances a spike needs, because instances become useful sooner.
  3. --min-instances sized to baseline RPS so a spike does not start from zero warm capacity. This costs money every hour, so size it from measured baseline, not from the peak.
  4. --max-instances set from peak RPS divided by concurrency, with headroom, and set last.

Also: if Pub/Sub publishing happens after the HTTP response is sent, set CPU always-allocated, or the CPU is throttled and latency leaks out of nowhere.

2. Design the release and rollback process to meet twice-daily deploys with sub-minute rollback, without keys in the pipeline.

Read the worked answer

Release: Cloud Run revisions are immutable and traffic is split by percentage, which gives you canary for free. Deploy with --no-traffic --tag=candidate, run smoke tests against the tagged URL, then shift 5% -> 25% -> 100%, checking error rate and p99 at each step. A tagged URL is the underrated part: it lets you test the new revision in production shape without exposing it to users.

Rollback: shift traffic back to the previous revision. It takes seconds, needs no rebuild, and works even if the registry is unavailable. That is the only mechanism that reliably meets under a minute.

Wrap both in Cloud Deploy with a canary strategy and an approval gate. A sequence of hand-typed gcloud run services update-traffic calls is not a release process and will not survive the person who wrote it going on holiday.

The caveat nobody plans for: rolling back code does not roll back schema. Require expand/contract migrations: add the new column, deploy code that writes both, backfill, then drop the old column in a later release. Otherwise the previous revision is no longer runnable against the current database and your sub-minute rollback is fiction.

Keys: none are needed. The Cloud Build service account runs the trigger inside your project; grant it roles/run.admin and roles/iam.serviceAccountUser on the runtime service account, and deploy with --service-account. For any external system, use Workload Identity Federation rather than a JSON key.

3. A second team wants to deploy from GitLab to a project in a different organization. What identity model do you give them, and how do you stop the next leaked key?

Read the worked answer

Use Workload Identity Federation, not a downloaded key. GitLab (like GitHub) can present an OIDC token, so:

  1. Create a workload identity pool and an OIDC provider trusting GitLab's issuer.
  2. Map attributes, and grant roles/iam.workloadIdentityUser on the deploy service account to the federated principal with an attribute condition pinning the assertion to that specific repository and ref. Without the condition, anyone who can get a token from that issuer can impersonate the service account, which is worse than a leaked key because it is invisible.
  3. Exchange the token for a short-lived credential in the pipeline; there is no long-lived secret to leak or rotate.

Scope the service account narrowly: roles/run.developer on the specific service plus roles/iam.serviceAccountUser on the runtime account, and nothing at project level.

Stopping the next one: set Org Policy constraints/iam.disableServiceAccountKeyCreation at the organization node. It is the only control that reliably prevents a well-meaning engineer from taking the shortcut under deadline, and it costs nothing today. Pair it with a policy that keys already in existence are reviewed, because the constraint does not retroactively delete them.