PCCP - Professional Cloud Certified Professional: Provisioning at Scale: Terraform Operating Model for 40 Projects

Separate blast radius from reuse

Use a platform repository for shared modules and organization-level configuration, and allow team repositories to consume versioned modules. Do not put all 40 projects in one state file: a bad plan could affect every project and every team would block on the same lock.

Suggested structure

• modules/project-factory: project creation, labels, billing link, default service accounts, audit logging, org policy attachments.
• modules/network: Shared VPC host/service attachment, subnets, firewall policy, private service access.
• modules/iam-baseline: groups, predefined roles, IAM conditions, deny policies, service account impersonation.
• modules/workload-baseline: Cloud Run service, Artifact Registry, Cloud Build trigger, alert policies.
• envs/org: folder and org policy assignments.
• envs/folders/<folder>: folder-level IAM and policy.
• envs/projects/<project>: one state per project or per small project group.

State and ownership

Store state in a versioned GCS bucket with CMEK, access logging, and object versioning. Use separate prefixes per environment and per project group. The platform team owns modules and org-level state; product teams own their project stacks and can propose module changes through pull requests. Pin module versions by Git tag. Shared VPC host configuration stays with the platform team; service projects consume outputs rather than editing the host.

Why this shape

It gives teams self-service through a project-factory module while keeping organization policy, Shared VPC, and audit logging centralized. It also limits the damage of a bad apply and makes drift detection practical.

Pull-request workflow

1. Engineer opens a PR against a team stack or a project-factory YAML file.
2. Cloud Build trigger runs terraform fmt -check, terraform validate, and terraform plan -out=tfplan using a deploy service account scoped to the target folder or project.
3. Policy checks run in the same pipeline: org policy simulation where available, OPA/Conftest rules for required labels, approved regions, public IP denial, and mandatory logging. Fail the PR on violations.
4. Plan output is attached to the PR for review. A platform reviewer approves only high-risk changes such as org policy, Shared VPC, or IAM at folder level. Standard project changes can use automated approval if they match the template.
5. On merge, the same saved plan is applied, not a newly generated plan. This prevents unreviewed drift between plan and apply.

Identity and approvals

Use Workload Identity Federation from GitHub to Cloud Build or directly to a deploy service account. No service account keys. The deploy account should be able to impersonate narrower per-folder deploy accounts, not own the organization. Cloud Build approvals or GitHub environment protection provide the human gate.

Keeping lead time low

Pre-approve the common path: a new project from the factory module with standard labels, logging, and a Shared VPC attachment. Automate policy checks and plan generation. Reserve manual review for exceptions. Run scheduled terraform plan drift detection so teams fix drift before it becomes a nine-day ticket.

Inventory before import

Start with Cloud Asset Inventory and labels to identify unmanaged resources. Prioritize by risk: public buckets, firewall rules, overly broad IAM, and service account keys. Do not import everything at once; group resources by team and environment.

Import safely

Use Terraform import blocks with terraform plan -generate-config-out where supported, then review and refactor the generated configuration into modules. For resources that cannot be imported cleanly, recreate them through the module and migrate traffic or data with a tested plan. Keep a read-only audit period where the pipeline reports drift but does not auto-remediate. Once a resource is owned, enable scheduled drift detection and require changes through PRs.

Service account keys

Treat the 18 keys as an incident, not a cleanup ticket. Inventory owners and usage, create replacement credentials using Workload Identity Federation or service account impersonation, deploy the change, verify, then disable old keys before deleting them. Apply an org policy to disable new key creation except for a break-glass project. Never store keys in Terraform state or Git.

Avoid a freeze

Publish a migration schedule by team. Offer a fast path for new projects through the factory module immediately, and a 30-day window for existing resources with platform support. Teams that need an exception can use a time-bound, reviewed exception with compensating controls. This keeps delivery moving while reducing the unmanaged surface.