Peregrine Logistics: A $1.35M Monthly Bill and a Plan to Buy 3-Year All-Upfront Everywhere
Peregrine Logistics spends about $1.35M per month. It runs 620 EC2 instances, 71% of them m5.2xlarge on 24/7 at 12% average CPU, which also execute 9,000 nightly batch jobs. Storage is 180 TB of gp2 EBS and 4.2 PB of S3, where 88% of bytes have not been read in 200 days but a regulator requires seven-year retention with retrieval within 24 hours. Data transfer out to the internet is $95,000 per month, NAT Gateway processing is $28,000, and app-to-database traffic generates 300 TB per month of inter-AZ transfer. Peregrine already holds one-year no-upfront Standard Reserved Instances covering 30% of the fleet. Engineering proposes buying three-year all-upfront Reserved Instances for 100% of the fleet to lock in maximum discount before a Graviton migration begins next year.
1. Critique the proposed commitment purchase and specify what Peregrine should buy, in what order, and at what coverage.
Read the worked answer
Three things are wrong with the proposal
- Committing before right-sizing commits you to waste. 620 instances at 12% average CPU on m5.2xlarge means the fleet is roughly 3x oversized. A three-year all-upfront purchase on today's footprint locks in the overspend for three years, with 100% of the cash paid up front and no exit.
- Not everything is steady-state. The nightly batch fleet is spiky by definition. Committing capacity to batch is paying reserved prices for idle hours.
- It conflicts with the stated plan. 100% EC2 Instance Savings Plans or Standard RIs lock instance family and region. A Graviton migration to m7g breaks coverage on precisely the assets you just committed to for three years. You cannot sequence a migration after a three-year lock-in.
Correct order of operations
- Right-size and consolidate the 24/7 fleet using Compute Optimizer and real utilisation data. Target m6i/m7g equivalents at roughly one-third of current vCPU.
- Move the batch fleet to Spot with checkpointing, diversified instance families, and capacity-optimized allocation. Batch is the ideal Spot workload and should not be in any commitment.
- Only then commit - to the post-right-sizing, post-Graviton steady baseline.
What to buy Compute Savings Plans as the backbone: up to 66% discount, and crucially they apply across instance family, region, OS, and tenancy, and cover Fargate and Lambda. Slightly less discount than EC2 Instance Savings Plans, but the flexibility is exactly what a Graviton migration requires. Ladder the terms: three-year no-upfront or partial-upfront on the irreducible core, one-year no-upfront on the next tranche so it can be re-evaluated.
Coverage target About 70% of the mature baseline, never 100%. The remaining 30% keeps elasticity and absorbs change.
On the existing Standard RIs Standard RIs cannot be exchanged for Savings Plans; they can be sold on the RI Marketplace, and they can be modified within family and region. Model the resale against the discount gain before assuming they are stranded. Convertible RIs, if any exist, can be exchanged for Savings Plans.
2. Address the storage and data-transfer lines: S3 lifecycle, EBS, and the $95k transfer-out plus $28k NAT bill. Show which lever moves the most money.
Read the worked answer
S3 is the biggest single lever
4.2 PB is roughly 4.3 million GB. At S3 Standard $0.023/GB-month that is about $99,000 per month today.
The 24-hour retrieval SLA is the crux: it rules out nothing cheap. Glacier Flexible Retrieval standard retrieval completes in 3-5 hours at $0.0036/GB-month ($15,500). Glacier Deep Archive completes in 12 hours at $0.00099/GB-month ($4,300) and still complies with a 24-hour SLA.
The judgement: with 88% of bytes unread in 200 days and a regulator who retrieves rarely, Deep Archive on the 88% and Glacier Flexible Retrieval on the remainder is defensible. But watch the gotchas that eat the savings: 180-day minimum storage duration for Deep Archive (early deletion is charged in full), 128 KB minimum billable object size, and $0.02/GB retrieval. If the corpus is many small files, aggregate them into archive packages before transitioning. If any meaningful fraction gets restored regularly, Glacier Flexible Retrieval wins on total cost despite the higher storage rate.
EBS 180 TB of gp2 at $0.10/GB-month is $18,000. gp3 is $0.08 and includes 3,000 IOPS and 125 MB/s baseline, so the migration alone saves $3,600 per month with no performance change - and gp3 lets you provision IOPS and throughput separately instead of over-sizing volumes to buy IOPS. Add deletion of unattached volumes and a snapshot lifecycle into the archive tier for anything held beyond 90 days.
Network
- 300 TB inter-AZ at $0.01/GB each direction is about $6,100 per month. Reduce by co-locating chatty read replicas and cache tiers, but do not remove AZ redundancy to save it - that trades availability for a rounding error.
- NAT Gateway at $0.045/GB is $28,000 because traffic is going through NAT to reach S3 and DynamoDB. Gateway endpoints are free and typically eliminate the majority of NAT-processed bytes. This is the highest return-per-hour-of-effort change on the entire bill.
- $95,000 transfer-out: front the static and cacheable content with CloudFront, negotiate private pricing at this volume, and confirm what is being served at all - a meaningful share of egress bills is unversioned assets fetched repeatedly.
Ranking
S3 tiering ($80k/month) far exceeds everything else. Then commitments after right-sizing, then NAT endpoints ($15-20k), then EBS (~$4k). Anyone who starts with instance-level cleanups spends a quarter for a rounding error.
3. Design the cost-control operating model, and explain the sequencing trap that would undo the savings within two quarters.
Read the worked answer
The sequencing trap Right-size, then migrate to Graviton, then commit. Reversing any pair of those costs real money: committing before right-sizing locks in 3x overspend; committing to EC2 Instance Savings Plans before Graviton strands the coverage; and doing either before the NAT and S3 work leaves the two largest lines untouched. Engineering's proposal inverts all of it, and the mistake is unrecoverable for three years.
The operating model
- Visibility: Cost and Usage Report 2.0 into S3, queried with Athena, published to per-team dashboards. Showback, not chargeback, in the first two quarters - you want behaviour change before budget fights.
- Unit economics, not totals. Track cost per 1,000 shipments alongside absolute spend. Total spend can fall while unit cost rises during growth, and the reverse. Unit cost is the only metric that survives a volume change.
- Tag enforcement. An AWS Organizations tag policy that denies creation of billable resources without Owner and CostCenter. Untagged spend is invisible spend, and tag hygiene is the control that always decays silently.
- Commitment governance. A central FinOps function owns all Savings Plans and RIs. No team may purchase commitments against its own account - that is how an organisation ends up with fragmented, overlapping, unusable coverage.
- Guardrails with teeth: AWS Budgets with forecasted-spend alerts, Cost Anomaly Detection wired to the owning team's on-call, and a monthly anomaly review with a named owner and a decision deadline, not an inbox.
- Architecture-level controls: SCPs or Service Catalog constraints that prevent provisioning in expensive regions, prevent over-sized instance families by default, and require an exception for anything outside the approved catalogue.
The discipline that keeps it from regressing A quarterly review of the same four numbers: commitment coverage against the mature baseline (target ~70%), Savings Plan utilisation (target above 95%), untagged spend percentage (target under 2%), and unit cost trend. A one-time clean-up without that cadence regresses to the original bill in two quarters - the cost problem is an operating-model problem, and the model is what has to change.