A $3,100 Bill and Nine Instances Powered Off
A team runs 22 Compute Instances and one LKE cluster. The monthly compute invoice is $3,100 against a $2,000 budget. To save money, an automation script powers off 9 staging and demo instances at 19:00 and back on at 07:00 on weekdays, and the team expects about 45% off those instances' cost. The invoice barely moved. Meanwhile 14 instances use Backups; three "temporary" 4 TB Block Storage volumes from a migration two months ago are still attached; two instances run GPU plans used four hours a week for model evaluation; and every deploy is done by SSH-ing in and running a hand-edited shell script, so nobody is sure which instances differ from the base image. A junior engineer proposes deleting all powered-off instances to cut the bill.
1. Explain why the power-off automation saved almost nothing, and what actually reduces the bill. Include a rough monthly figure for the idle 4 TB volumes.
Read the worked answer
Why it failed: Compute Instances bill hourly until the plan's monthly cap, and a powered-off instance still exists and is still billed. Powering off stops the guest, not the charge. Only deleting the instance ends the cost. So the script changed the state of nine machines without changing the invoice.
What actually moves the number:
- The three idle volumes: 12,000 GB x $0.10 = $1,200/month for data nobody reads. Verify contents, snapshot if required, then delete. This single action is larger than the entire savings the script was supposed to deliver.
- Stopped instances: delete and rebuild from a defined image when needed.
- GPU plans: billed whether or not a notebook is running; four hours a week on a monthly plan is the worst ratio in the account. Either schedule the work into a short-lived instance created on demand, or accept and budget for it explicitly.
- Backups: 14 of 22 instances, including demo boxes; retention and coverage should follow recovery requirements, not default to on.
- Right-sizing after you know actual utilization.
Set a budget alert, or this happens again next quarter.
2. Evaluate the junior engineer's deletion proposal: what is the risk, and what is the correct sequence?
Read the worked answer
The instinct is right; the action is reckless. Deleting an instance destroys its disks unless volumes are detached or backups restored. Staging boxes are where undocumented configuration, unmerged branches, and test fixtures live - precisely the state that is not in version control. Deleting first converts a cost problem into an incident.
Sequence:
- Inventory by tag and owner; anything untagged is a finding in itself.
- Verify backups or images exist and actually restore, tested on one sacrificial instance.
- Notify owners with a deadline, then remove DNS records and firewall rules so nothing silently depends on the host.
- Delete and confirm the invoice line disappears the following cycle.
- Prevent recurrence with mandatory tags, a demo/staging lifecycle policy, and a monthly unused-resource review.
Deleting is the last step, not the first; the deliverable is an inventory plus a tested rebuild path.
3. Replace the SSH-and-shell deploys. Compare StackScripts, custom Images, Terraform, and Marketplace apps, then pick a combination.
Read the worked answer
These solve different layers, and confusing them is the classic mistake.
- StackScripts run at first boot to customize a host. Good for small, idempotent bootstrap steps; poor as a full configuration-management system.
- Custom Images capture a known-good disk, including the application runtime. This is how you kill configuration drift.
- Terraform (Linode provider) declares infrastructure - instances, volumes, firewalls, NodeBalancers, LKE - and gives you a diff and a review before anything changes.
- Marketplace apps deploy third-party software stacks quickly. Useful for a standard tool, wrong for a proprietary application.
Recommended combination: bake the application into a custom Image; define all infrastructure in Terraform with tags and state in shared storage; use a short StackScript only for per-boot environment injection; drive it from CI so a merge, not an SSH session, produces a deploy. Then rebuild the 22 hosts from the image to eliminate drift.