A Three-Point-Four Times Cloud Bill and an Untested RTO

Do the cheap, reversible cost work first, and do not trade resilience for cost before the failover has been proven.

Order of operations.

1. Find the money that costs nothing to reclaim. Orphaned volumes, unattached IPs, idle non-production estates, and forgotten snapshots typically account for a large slice of a 3.4x overrun. No performance risk.
2. Tag and show back. You cannot manage what cannot be attributed; a week of tagging discipline changes behaviour before any instance is resized. No tag, no production deploy.
3. Commit the stable baseline. Purchase reserved or savings-plan capacity for the portion of load that is provably flat - that is where most of the discount lives, and it does not constrain elasticity for the bursty tier.
4. Right-size with a guardrail. 62% idle by CPU is not proof of over-provisioning - CPU is a poor proxy for memory-bound and I/O-bound hosts. Check P95 memory and I/O before shrinking, and pilot on one host at a time.
5. Separate batch from interactive onto distinct capacity so the 22:00 job stops forcing the interactive tier to over-provision.

Not tradeable: the SLA. 99.95% is about 21 minutes of downtime a month. A single-region design can plausibly meet that against instance and availability-zone failures, but only if failover actually works. Prove multi-AZ failover before optimizing cost. Multi-region is a separate, expensive question that should be answered with the SLA arithmetic and the customer contract, not by default.

Design the test to be honest. State the hypothesis first: given loss of the primary region (or zone, define which), the customer-facing service is restored within 4 hours and loses no more than 15 minutes of data. Then test against that, not against a scripted walkthrough.

Run it in stages: a dependency-mapping exercise first, because the 11-hour answer is usually hiding in a forgotten dependency - DNS TTLs, a licence server, an identity provider, a vendor connection that has never been re-pointed. Then a non-production restore, then a production game day with a defined customer impact window and comms. Measure restore time from the actual start of the outage, and measure replication lag at cutover to get the real RPO.

If it measures 11 hours, do not report four. Report the measured figure, the gap, and the cost of closing it. Three legitimate responses: invest to close the gap (automated failover, pre-warmed standby, shorter TTLs); renegotiate the SLA and the DR attestation to what is actually achievable; or formally accept the gap with an executive signature, an expiry, and a funded plan.

What is not legitimate is the annual attestation asserting 4 hours in six weeks' time when you now know otherwise.

Flapping between minimum and maximum several times a day almost always comes from scaling on the wrong signal with too little damping.

Likely causes. A CPU-threshold policy reacting to a signal that moves with the batch job rather than with user load. No cooldown or a cooldown shorter than instance boot time, so the group reacts to a metric that still reflects the pre-scale state - a feedback loop. Instance warm-up longer than the scaling interval. A maximum size that is too low, so the group pins at max, cools, drops, and repeats.

Fixes. Scale on the metric that tracks user pain - request queue depth, latency, or request count - with a target-tracking policy rather than a step threshold. Set the cooldown above boot-plus-warm-up. Use pre-baked images or a warm pool so a new instance is useful in under a minute.

What it reveals. The deeper fault is that the batch workload and the interactive tier share a capacity pool with different demand curves, so each fights the other. Split them onto separate pools and the interactive tier's baseline becomes predictable - which is also what makes reserved capacity safe to buy and the 3.4x bill tractable. Flapping is expensive in itself: you pay for instance-hours that serve no requests, and you add churn risk to the SLA you are already not measuring.