One shared project, one billing account, two years of drift

Hierarchy: Org -> Prod and Org -> Non-prod, each containing one folder per engineering group, each containing that group's projects. Split on environment first because the controls that matter (who may create projects, who may touch production) are per-environment and IAM/Org Policy inherit downward; putting group at the top would mean attaching production controls four times and eventually forgetting one.

Controls:

• roles/resourcemanager.projectCreator on each Non-prod group folder only. Self-service with no ticket, and no ability to mint a production project.
• Production projects are created by a pipeline running as a service account. Humans get no Owner in Prod; deploy identities get narrow roles. This is the control that actually kills the incident: the staging identity has no grant anywhere in the Prod subtree, so it cannot delete a production subscription regardless of what the code does.
• The second-approval requirement lives in the release pipeline (a Cloud Deploy approval gate), not in IAM. IAM has no native two-person rule for a single action; do not pretend a conditional binding gives you one.
• constraints/iam.allowedPolicyMemberDomains at the Org node blocks the gmail.com grant.
• Shared VPC host project under a Network folder; roles/compute.networkUser granted on the group folders so workloads attach to subnets without owning the VPC.

Why not labels: labels are for billing and inventory queries. They carry no enforcement, so env=prod on a project stops nothing.

Grant to a Google Group, never the individual, and put the expiry on group membership in Google Workspace. IAM Conditions with request.time work on many resources but not uniformly, and a group whose membership auto-expires is one mechanism that behaves the same everywhere.

Before:

• Confirm the domain-restriction Org Policy is actually enforced at this project (it is what makes the external address usable at all).
• Check for indirect paths that make the grant broader than intended: is the dataset authorized to a view in another project, and do nested groups already hold Editor?
• Use Policy Analyzer (gcloud asset analyze-iam-policy) to see effective access rather than reasoning from the bindings you can see.

Grant: roles/bigquery.dataViewer on the dataset resource (not the project), plus roles/bigquery.jobUser if they need to run queries and bill the cost to their own project.

After:

• Confirm the change in Admin Activity audit logs.
• Enable Data Access logs for BigQuery for the six weeks if the auditor wants read evidence; note these are high-volume and are off by default, so enable them with intent, not by default.
• Diarise the removal. Access that expires on paper but not in practice is the most common audit finding.

Quotas are scoped per project, per region, per metric, so identify which one fired before doing anything. In practice it is one of: total SSD disk capacity, total disk count, or in-use IP addresses, and frequently the job is also leaking disks from earlier failed runs. Check the project's Quotas page and gcloud compute regions describe us-central1 for the region's quota status, then list disks with no users to find orphans.

Immediate: request an increase for that specific metric and region. Increases are not instant and not retroactive, so rerun on standard PD or in a region that still has headroom if the deadline is real.

Structural, in order of value:

1. A Monitoring alert on the quota usage metric so you find out at 80% instead of 100%.
2. A cleanup job for unattached disks, run daily.
3. Know that a new project starts at default quota and a new region has its own independent quota, so the project factory must request baseline increases as part of creation, not after the first failure.
4. If the pipeline is region-agnostic, make region a parameter so a quota wall in one region is a retry, not an incident.