Reliability Engineering, Capacity, and Performance
Reliability Engineering, Capacity, and Performance
This domain tests load testing, capacity headroom, autoscaling, and failure-mode design.
Core rules
- Capacity planning starts with demand forecasts and performance budgets, not average utilization. Plan for peak and for failure of one zone or node.
- Use load tests that model real traffic mix, think time, and dependency latency. Measure p95 and p99, not only averages.
- Set Kubernetes resource requests and limits; use Horizontal Pod Autoscaler for replica scaling and cluster autoscaler for node capacity. Code Engine scales serverless workloads.
- Design for graceful degradation: timeouts, retries with backoff, circuit breakers, queue-based load leveling, and load shedding.
- Run chaos experiments and failover tests to validate assumptions and improve MTTR.
Worked example
A service runs on IBM Cloud Kubernetes Service with a 70% CPU target for HPA. A load test at 2× peak shows p95 latency rising before CPU reaches 70% because a downstream database is saturated. The team adds a connection pool limit and a read replica, then retests. Reasoning: autoscaling pods cannot fix a bottleneck in a shared dependency; capacity must be validated end to end.
Traps
Scaling on average CPU hides hot pods. HPA without resource requests cannot compute utilization. Testing only the happy path misses retry storms. Ignoring dependency quotas causes cascading failure. Overprovisioning without cost control is not capacity planning.