Cost Anomaly, Untagged Spend, and a Commitment Decision

Order by reversibility and by whether the fix needs data to be correct.

1. Containment (days) - fix without waiting for data. The sandbox has no guardrails, so the same event can recur tomorrow. Set budget alerts at account and team level with notifications to a named owner, restrict GPU and other high-cost instance families by policy so they require an approved exception, and enforce auto-shutdown and maximum lifetime on non-production resources. Enforce it technically at the organisation level; a memo is not a control.
2. Allocation (weeks) - this is what blocks everything else. Require owner, cost centre, environment, and service tags at resource creation, and backfill by mapping legacy resources to an account-level default. Treat the remaining unallocated balance as a visible line item owned by a named executive, not as rounding. Consider restructure so the account boundary itself is a cost boundary, which works even when tags fail.
3. Unit economics and optimisation (weeks). Rightsizing, storage tiering, and schedules on the steady-state and batch workloads, measured as cost per transaction or per 1,000 API calls.
4. Commitments (last) - these must wait. Purchasing savings plans in week one would lock in a baseline distorted by the GPU anomaly, converting a variable cost into a fixed one for a workload that may be re-architected.

The forecast miss and the anomaly are the same disease: no guardrails, no allocation, no early warning.

Establish a clean baseline first

Strip the anomaly out. If the GPU spike accounted for, say, $40,000, the true-monthly base is about $170,000, and 60% steady state is roughly $102,000.

Coverage target

Commit to about 70–80% of the verified steady-state baseline, roughly $70–80k/month. Leave the spiky 40% on demand - that portion is exactly what a commitment cannot absorb, and over-committing is the classic failure here: you keep paying for capacity after the workload is re-architected or migrated.

Term and payment

• One-year term, no-upfront or partial-upfront, and a flexible or convertible instance family. The workloads have been stable for two years, but a two-year track record is not a guarantee that the platform roadmap will not change. Flexibility has a price, and at this scale it is worth paying.
• Three-year, all-upfront only for the subset with no migration or replatforming decision pending - and only after those decisions are made. For a fintech, cash preservation usually outweighs the extra discount.

Governance

• Review coverage monthly; alert when coverage falls below 60% or rises above 90%.
• Align commitment expiry to the roadmap so it is a deliberate renewal rather than an auto-renewal.
• Report the discount achieved against on-demand as a tracked metric, so the commitment is defended with numbers, not assumed.

Why they failed

• The data arrived late and was a third unallocated, so no line could be attributed to a decision-maker.
• There was no denominator. A dollar figure without cost per transaction or per customer gives engineers nothing they can act on.
• It was a report, not a control: no target, no owner accountability, no consequence, no review.
• A monthly spreadsheet has no interrupt. The GPU event would have gone unnoticed until the invoice either way.

Design that sticks

• Allocation first. Get to 95%+ allocated before publishing anything. Show the unallocated residual explicitly, with an owner.
• Push, don't publish. Anomaly alerts and weekly run-rate versus budget go to the owning team's channel, with the resource and the tag in the message.
• Denominators. Cost per transaction, per active customer, per 1,000 API calls - a metric the team can move.
• Forecast as a first-class output. The CFO's concern was the miss, not the total. Show run-rate and projected month-end, and alert when the projection drifts.
• Attach levers. Each team's target comes with the controls it can pull: rightsizing, storage tiering, shutdown schedules, commitment coverage.
• Close the loop. A 30-minute monthly review where the top three variances are explained by their owners. Chargeback only after showback is trusted; billing teams for costs they cannot influence destroys credibility.