Untagged test environment and a 40% bill spike

Start with a small mandatory tag schema and enforce it at resource creation.

• Required tags: team, environment, owner, purpose, and cost-center. team maps to the product team, environment separates production, staging, and test, and owner identifies a named person or group. purpose distinguishes a permanent workload from a proof of concept.
• Enforcement: Use a policy that denies creation of compute, storage, and load balancer resources without the required tags. If the platform cannot deny at creation, run a daily report of untagged resources and require the owning team to tag them within one business day.
• Chargeback mapping: Build a monthly report that groups cost by team and environment. Shared costs such as DNS or a central load balancer should be allocated by a documented rule, for example proportional to each team's compute spend, rather than left unallocated.
• POC lifecycle tag: Add a expires-on tag for temporary environments. The POC that launched 12 servers and 20 TB should have had an expiry date and an owner who received a reminder before it lapsed.
• Orphan detection: Compare the resource inventory against tags and known deployment stacks. Untagged or expired resources are reported to the platform team for review, not silently deleted.

The key judgement is that chargeback is a data-quality problem before it is a finance problem. Without a mandatory, enforced tag schema, the next bill spike will be just as unexplainable.

The current monitoring is too shallow for both cost and reliability.

Cost detection:

• Set budget alerts at 50%, 80%, and 100% of the expected monthly run rate, and a separate anomaly alert for a day-over-day increase above a threshold. The 40% spike should have triggered before the invoice was issued.
• Break alerts down by team and environment once tagging is enforced. A single account-wide alert tells finance that something happened but not who owns it.
• Add a weekly report of new resource creation by team and resource type. A POC that creates 12 servers and 20 TB should appear as an unusual creation event, not only as a monthly total.

Incident detection:

• Replace the single ping check with checks for CPU, memory where available, disk, network, and application response time. A ping only proves the host is reachable; it does not show that the application is degraded.
• Centralize application logs and extend retention beyond seven days. A four-day diagnosis suggests the team needed logs from before the incident window. Set retention based on the incident-response requirement, not the development convenience.
• Add alerting on error rate and latency, not just availability. The customer impact may occur while every ping succeeds.
• Define an on-call path and a runbook for the top alerts so a small team can respond without a large operations staff.

The design principle is to alert on the signals that precede the bill and the outage, not only on the final symptom.

Use lightweight, automated controls that make cleanup the default rather than a manual reminder.

1. Require an expiry date at creation for test and POC environments. A expires-on tag or a request form field should be mandatory. The default can be 14 or 30 days, with an explicit extension.
2. Send automated reminders to the owner at 3 days and 1 day before expiry, with a one-click extension path. The owner can extend without filing a ticket, which keeps the process fast.
3. Quarantine rather than instantly delete. At expiry, stop or isolate the resources and notify the owner. If there is no response after a defined grace period, delete them through an approved workflow. This protects against accidental loss of useful work.
4. Show cost in the team's own report. When a team sees its POC spend in its monthly chargeback, it has an incentive to clean up. Cost visibility is a process control, not just a finance artifact.
5. Make the approved deployment path the easiest path. Provide a small template or script that creates a tagged test environment with an expiry date. If the sanctioned path is slower than ad hoc scripts, teams will bypass it.
6. Review exceptions monthly. The platform team reviews expired resources, untagged resources, and repeated extensions. Trends indicate whether the default expiry is too short or whether a team needs a permanent test environment.

The key judgement is to design for the normal case: most teams will clean up if the system asks them at the right time and makes extension easy. The control should catch the minority who forget, not punish everyone with a slow approval process.