36 Projects, Local State, and a $130k Spend Increase
Your company has 36 cloud projects: 4 production, 12 staging, and 20 sandbox. Infrastructure is created with a mix of Terraform, console clicks, and vendor scripts. Terraform state is stored on engineers' laptops; the last audit found 19 untracked production resources and one state file with plaintext credentials. Monthly spend grew from $210k to $340k in five months. Finance sees no consistent labels, so cost allocation is manual. Sandbox projects have no expiration, and 140 unattached disks plus 60 idle load balancers were found. A new policy requires every resource to have owner, cost-center, and environment labels within 30 days, and requires infrastructure changes to be reproducible and reviewed. The platform team has five engineers and already supports 200 developers.
1. Design the infrastructure-as-code operating model. Address state, untracked resources, drift, and the review requirement without creating a bottleneck.
Read the worked answer
State. Move Terraform state to a remote backend with encryption, versioning, and locking. Never keep state on laptops. Treat state as sensitive because it can contain credentials; rotate anything exposed in the leaked file and audit access. Split state by environment and by lifecycle, not one giant state, so a sandbox change cannot threaten production.
Workflow. Pull requests run terraform plan and policy checks; merges run apply from a pipeline with a scoped identity. No console changes for production except break-glass. The pipeline records the plan, approver, commit, and apply result.
Untracked resources. Import the 19 resources into state rather than deleting them. For each, confirm ownership and labels, then write matching configuration. Do not run a broad apply before import; it may propose destructive changes.
Drift. Schedule a read-only plan and alert on drift. Fix drift by changing code or, for emergency changes, by codifying the change immediately.
Why this fits. Five engineers cannot hand-manage 36 projects; the pipeline enforces reproducibility and review without adding a gate for every resource.
2. Design the label and governance guardrails. How would you enforce owner, cost-center, and environment without breaking existing automated deployments?
Read the worked answer
Label policy. Define a small required set: owner, cost-center, environment. Enforce with policy-as-code at creation time for production and staging; for sandbox, start with warning mode and a 30-day grace period. Default labels in reusable modules so developers do not have to remember them. Labels must propagate to resources that support them; where they do not, use a mapping table or a cost-allocation export.
Prevent drift. Run a daily compliance report and a weekly remediation queue. Do not rely on Finance to find unlabeled resources after the invoice. Block new production resources without labels after the grace period; allow a documented exception with an expiry date.
Ownership. Every project needs an owner group, not an individual. Cost-center must map to Finance's chart of accounts, or allocation will fail again.
Evidence. Policy violations trend, percentage of spend with complete labels, and exceptions with expiry.
Tradeoff. Hard enforcement can break automated deployments that create resources outside the module. Provide a supported module and a fast exception path; otherwise teams will work around the policy.
3. Build a cost-reduction and lifecycle plan. Prioritize actions and explain how you would prevent the savings from eroding.
Read the worked answer
Attack the known waste first. For the 140 unattached disks, snapshot if retention is required, then delete; confirm they are not part of a disaster-recovery process. For the 60 idle load balancers, check traffic over at least 30 days and remove those with none. Automate a weekly orphan report.
Sandbox lifecycle. Put sandbox projects on a TTL, for example 90 days, with an extension request. Notify owners before expiry and archive data if needed. This controls the fastest-growing segment.
Visibility and accountability. Use labels to produce showback by cost center. Set budget alerts at 50/80/100% and anomaly alerts on daily spend, but route them to owners, not just Finance. The jump from $210k to $340k needs a driver analysis: separate usage growth from price and waste.
Commitments. Only after removing waste and understanding baselines, buy commitments for steady production usage. Do not commit to sandbox.
Targets and governance. Publish a monthly waste number and an owner-level scorecard. Without an owner, cost work becomes a periodic cleanup instead of a control.