Error Budgets and Incident Practice at a Logistics SaaS
ParcelPilot runs a multi-tenant logistics SaaS on 23 microservices across two regions. Its public availability target is 99.95%, but last quarter it achieved 99.91%. Major incidents rose from 9 to 15; mean time to detect is 45 minutes and mean time to restore is 4 hours. Change failure rate is 22%, with 12 deployments per week. One on-call engineer covers all services; post-incident reviews assign blame to individuals. Dashboards contain 47 infrastructure metrics, but no service-level indicators are agreed with customers. The CTO wants to add a third region and double release frequency; the COO wants fewer outages; the finance director questions the cost of resilience. Two enterprise customers have invoked contractual credits. The error budget is tracked by no team.
1. Diagnose the primary constraint on reliability at ParcelPilot. Explain why adding a third region and doubling deployments may worsen outcomes, and what should precede both.
Read the worked answer
The primary constraint is not region count. It is the combination of weak detection, slow restoration, high change failure, and no agreed service-level indicators. A 45-minute mean time to detect and a 4-hour mean time to restore mean customers experience long outages even when infrastructure is redundant. A 22% change failure rate at 12 deployments per week implies roughly 2.6 failed changes weekly, so doubling release frequency without better change control would likely increase incidents. One on-call engineer for 23 services cannot provide deep coverage across all failure modes, and blame-oriented reviews suppress the information needed to fix systemic causes.
Adding a third region increases the operational surface: more deployment targets, more data consistency paths, more network and identity complexity, and more cost. With the current detection and change practices, it may add failure domains rather than remove them. It could be justified for a specific regulatory or concentration risk, but not as the first reliability investment.
Before either expansion, ParcelPilot should define customer-facing SLIs for order tender, tracking, and invoicing; set SLOs and error budgets; instrument those journeys; introduce progressive delivery and canary analysis; establish service ownership and blameless post-incident reviews; and create an on-call rotation with runbooks. The error budget should be visible to product and engineering and used to gate release velocity. The third region and doubled deployment frequency should be conditional on sustained error-budget compliance and improved change failure rate.
2. Design an error budget policy and a severity model that the CTO and COO can both accept.
Read the worked answer
The error budget policy should translate the 99.95% target into a monthly allowance of about 21.6 minutes of unreliability for each customer-facing SLO. Track burn rate rather than waiting for month-end: a fast-burn alert when 2% of the monthly budget is consumed in one hour, and a slow-burn alert when 10% is consumed in six hours. Measure at the customer edge and by tenant tier so that a single large customer or region does not hide behind an average.
The policy should state consequences in advance. When a service exhausts its budget, non-essential releases for that service freeze until the budget is restored or an executive exception is approved with a risk statement. Reliability work takes priority in the next sprint. When budgets are healthy, teams may increase release frequency within canary and rollback standards. This gives the CTO a path to speed and the COO a predictable control.
The severity model should be based on customer and contractual impact, not internal cause. Severity 1 covers loss of order tender or tracking for multiple tenants, data integrity risk, or a security breach; Severity 2 covers degraded performance or a single major tenant; Severity 3 covers limited workaround; Severity 4 covers no customer impact. Define response times, communication cadence, incident commander, and credit assessment for each. Post-incident reviews should produce owned actions with dates. The finance director can then see resilience spending linked to avoided credits and churn, not as an unquantified cost.