Rightsizing requests without breaking latency SLOs
A 1,200-deployment cluster costs $96,000/month; 68% of the bill is memory-optimized nodes. Requests, not usage, drive node count: total CPU requests are 48,000 cores against a 95th-percentile usage of 15,800 cores; memory requests are 150 TiB against a 95th-percentile of 72 TiB. The platform team proposes a one-time 50% cut to all requests to halve the bill. Three services are latency-sensitive ad-serving APIs with p99 SLOs and Java warm-up spikes; two batch pipelines run at 3 a.m.; one team already uses HPA on CPU at a 70% target. Last quarter a manual memory-limit reduction caused 14 OOMKills and a two-hour incident. Finance wants savings this quarter, and the platform team has two engineers for six weeks.
1. Evaluate the proposed 50% cut and estimate realistic savings. What would you do instead?
Read the worked answer
Reject the blanket cut. The arithmetic is attractive: memory requests at 150 TiB are 2.1× P95 usage, and halving them to 75 TiB still leaves about 4% headroom over P95. If memory requests fully drive node count, the memory-optimized portion ($65,280) could fall by up to half, about $32,600/month. That is an upper bound, not a forecast. Node fragmentation, DaemonSets, minimum node counts, and pod disruption budgets absorb part of it; a realistic first pass is 20–30% of the memory pool, or $13,000–$20,000/month.
The CPU cut is riskier. Requests fall from 48,000 to 24,000 cores, still above 15,800 P95, but HPA on CPU uses requests as its denominator. Lowering requests raises reported utilization and makes HPA scale out earlier, adding pods and cost. Java warm-up and p99 SLOs need burst headroom that P95 averages hide. The prior OOM incident also suggests requests and limits were not separated.
Instead, segment workloads: memory requests to roughly 1.3× P95 per workload, CPU requests to about 1.5× P95 for latency-critical services and 1.2× for batch, using VPA recommendations rather than one global percentage.
2. Design a six-week rollout with two engineers. What do you change first, and how do you protect the ad-serving APIs?
Read the worked answer
Week 1: enable VPA in recommendation-only mode and rank workloads by cost. The top 50 workloads usually carry most of the spend, so focus there. Build a dashboard of requests vs P95 usage and exclude the three ad APIs from automated changes.
Weeks 2–3: change memory requests first, because memory is the billing driver. Apply VPA recommendations at 1.3× P95, one workload at a time, with pod disruption budgets and canary checks on OOMKills and p99 latency. Do not change memory limits in the same step as requests; keep limits at or above the new request.
Weeks 4–5: adjust CPU requests for batch and internal services at 1.2× P95. For the ad APIs, use VPA for memory and keep HPA on a custom metric such as requests per second, not CPU, so the two controllers do not fight. Add HPA behavior controls to prevent flapping.
Week 6: consolidate node pools and verify actual node count, not just requested capacity. Report savings and exceptions. A six-week window is enough for a controlled first wave, not for all 1,200 deployments.
3. How do you prevent requests from creeping back up after the project ends?
Read the worked answer
Make requests a governed interface, not a deployment detail. Set a request-to-usage service-level objective: CPU requests no more than 1.6× P95 and memory no more than 1.4× P95, with documented exceptions for warm-up and latency-critical services. Enforce label and request defaults with admission policies and LimitRange defaults so a new deployment cannot ask for a full node by accident.
Charge teams on requests, not only usage, because requests drive provisioning. Publish direct and fully loaded views so a team can see the cost of its headroom, and keep idle capacity visible to the platform team. Run a monthly review of the top 20 workloads by request waste, with a named owner and a two-week response window. Add a regression check in CI that flags a request increase above a threshold against the last approved value.
The governance only works if exceptions are cheap to request and expensive to ignore. A short exception form with an expiry date is better than a policy that teams route around by deploying to a new namespace.