Streaming API Reliability: Active-Active Ambition Versus Practical SLOs
Streamline Media operates a video streaming API on a regional GKE cluster in us-central1. It serves 8 million monthly users and peaks at 40,000 requests per second across three services. The SLO is 99.95% availability, but a bad ConfigMap rollout caused a 22-minute outage last month and exhausted the quarterly error budget. The platform team deploys 12 times per day with kubectl apply and has no automated canary analysis. Stateful data lives in Cloud SQL for PostgreSQL and Memorystore for Redis. Leadership wants active-active multi-region within six months; finance has approved only a 25% infrastructure increase. The team has strong Kubernetes skills but no multi-region experience. Customers are concentrated in North America, with 10% in Europe.
1. Evaluate whether active-active multi-region is the right six-month target. Recommend a reliability architecture and explain the trade-offs.
Read the worked answer
Separate stateless from stateful
Active-active is realistic for the stateless API tier, but not for the whole stack within six months and a 25% budget increase. Cloud SQL has a regional primary and cross-region replicas; Memorystore is regional. True active-active writes would require a different data architecture, conflict handling, and operational experience the team does not have.
Recommended target
Use a multi-region active-passive design for stateful data and active-active for stateless front ends:
- Two regional GKE clusters, us-central1 and us-east1, behind a global external HTTPS load balancer with health checks and failover.
- Cloud SQL primary in us-central1 with a cross-region replica in us-east1. Document and rehearse promotion. Accept a measured RPO based on replica lag.
- Treat Memorystore as a cache, not a source of truth. Design cache-aside behavior so a regional Redis loss degrades performance but does not lose durable state.
- Keep Europe on the global front end initially; a third region is not justified by 10% of users and the budget.
Why not full active-active
Full active-active for writes would need a globally consistent database or application-level conflict resolution, plus runbooks and testing the team cannot build in six months. It would also likely exceed 25% cost increase. The active-passive stateful tier gives regional failure protection with a clear RTO, while the active-active front end removes the single-region compute failure.
Trade-off
Failover is not instantaneous, and the RPO depends on replication lag. That is acceptable if the business agrees to the measured RTO/RPO and the team rehearses promotion.
2. Design a deployment and rollback process that would have prevented the 22-minute ConfigMap outage. How do you make 12 deployments per day safer?
Read the worked answer
Replace imperative apply
Move to GitOps or a delivery pipeline that renders manifests from version control and applies reviewed changes. Do not let engineers run kubectl apply directly against production. Store ConfigMaps and Deployments in Git, use image digests, and require a pull request.
Progressive delivery
Use a canary or blue/green process with automated analysis:
- Deploy the new version to a small canary slice, for example 1% of traffic.
- Compare error rate, p95 latency, saturation, and a business metric against the baseline for a defined soak period.
- Promote to 5%, 25%, then 100% only if the metrics stay within thresholds.
- Automatically roll back to the previous ReplicaSet and ConfigMap if error rate or latency exceeds the threshold.
Config-specific safeguards
Validate ConfigMaps and environment variables in CI. Use schema validation or admission policies to reject malformed configuration. Separate configuration changes from image changes when possible. Keep the last known good ConfigMap version and make rollback a one-step, tested action. Add a pre-deploy check that the new config can start a pod and pass readiness checks.
Operational discipline
Freeze risky changes during peak events unless they are reviewed. Record every deployment and rollback in the incident timeline. After the outage, run a game day that deploys a deliberately bad ConfigMap to a staging cluster and proves automatic rollback works.
3. Define the SLO and error-budget policy for the API. What burn-rate alerts, incident practices, and release rules should the team adopt?
Read the worked answer
SLO math and scope
99.95% monthly availability allows about 21.9 minutes of downtime per 30-day month; quarterly it is about 65.7 minutes. Define availability from the user's perspective at the load balancer or API layer, not just pod readiness. Add latency and success-rate SLOs for key endpoints if availability alone hides user pain. Measure by service and by critical user journey.
Burn-rate alerts
Use multi-window burn-rate alerts rather than a single static threshold. Alert quickly on a fast burn that would exhaust a large portion of the budget in hours, and page on a slower burn that would exhaust it over days. Route the fast burn to the on-call engineer and the slow burn to the team's daily review. Avoid alerting on every 5xx; alert on budget consumption.
Error-budget policy
When the budget is exhausted, pause feature releases that affect reliability and prioritize reliability work until the budget recovers or the policy exception is approved by engineering and product leadership. Do not silently ignore the policy. The 22-minute outage should have triggered a review of the release process, not just a postmortem document.
Incident practice
Define severity levels, an incident commander role, and a communication channel. Use blameless postmortems with owned action items and due dates. Track action-item completion as a reliability metric. The goal is not zero incidents; it is fast detection, fast rollback, and learning that changes the system.