Loyalty SaaS: 38% Cost Growth and a 99.95% Availability Target

Start with allocation, then elimination, then optimization. A 38% spend rise against 12% volume growth means the problem is not just traffic. Enforce a tagging policy for owner, environment, service, and cost center, and use budgets and showback so three dev/test environments stop hiding in production. Look for orphaned volumes, idle virtual server capacity, unattached floating IPs, duplicated monitoring, and COS buckets with no lifecycle policy. The 400 days of logs are a clear candidate: keep a short hot retention for search, transition to a lower-cost class, and archive or expire according to policy. Right-size only after 30 days of utilization data; CPU alone is misleading, so use memory, network, and I/O. For Kubernetes, set requests and limits, use cluster autoscaler and horizontal pod autoscaling on meaningful metrics, and isolate batch workloads. Do not take a uniform 20% cut across production; that is how redundancy, headroom, and reliability get damaged. Separate fixed platform cost from variable workload cost so the CFO can see what is controllable.

Define SLOs for the points-balance API: availability, p95 latency, and error rate, with an error budget. 99.95% monthly allows roughly 22 minutes of downtime, so a single-zone cluster and an unconstrained batch job are incompatible with the target. Run worker nodes across three zones, use pod disruption budgets, readiness and liveness probes, and anti-affinity for critical services. Move the batch job to a dedicated node pool with taints and tolerations and network bandwidth limits, or schedule it off-peak; the two incidents point to resource isolation, not application bugs. Autoscale on requests per second, queue depth, and p95 latency rather than CPU. Add load shedding and circuit breakers for flash sales, and pre-scale before known events. Instrument the full path; 520 ms p95 may be a downstream dependency or a saturated node, not the API itself. Report SLO compliance weekly and use the error budget to decide whether to ship features or fix reliability. A 99.95% target also needs tested failover and a capacity plan for zone loss.

Both goals are possible only with a scope change. A 20% cut plus a reliability increase usually requires retiring or deferring something: old environments, duplicate tooling, over-retained logs, or non-critical features. Model three scenarios: cost-only (cut 20%, accept current reliability), reliability-first (cut 8-12%, fund multi-zone and isolation), and balanced (cut 12-15%, hit 99.95% by retiring dev duplicates and tiering logs). The balanced case is the honest recommendation. Protect the redundancy that supports the SLO: multi-zone nodes, backup, and observability. Cut waste, not resilience. If the VP insists on 20%, ask for authority to retire a service or reduce log retention below the current just-in-case policy, and document the risk. Put the cost and reliability targets on the same dashboard so trade-offs are visible, and review monthly against the error budget. This is the judgment the exam tests: cost optimization is not a uniform percentage cut, and availability is not free.