Vertex Pay: an exhausted error budget and a CTO who says SLOs are an SRE matter
Vertex Pay's payment API carries an SLO of 99.95% availability over a rolling 30-day window, which allows about 21.6 minutes of unavailability. By day 18 of the current window, two incidents have consumed 38 minutes. The agreed error budget policy states that when the budget is exhausted, non-critical releases are frozen until reliability work restores it, and the decision is jointly owned by product and engineering. The product owner wants to ship a feature next week that requires a database migration. The CTO says SLOs are 'an SRE concern' and that the business, not SRE, decides what ships. Meanwhile, SRE time is 65% toil: manual deployment steps, repetitive alert triage and access requests. Two engineers have resigned in six months, citing firefighting.
1. Explain the purpose of the error budget, who owns it, and how this conflict should be resolved.
Read the worked answer
The error budget turns the SLO into a shared decision currency: 99.95% over 30 days allows roughly 21.6 minutes, and 38 minutes means it is spent. Its purpose is not an SRE veto but a pre-agreed rule that lets product and engineering trade reliability against new scope using the same evidence; ownership is joint. If the CTO reframes it as an SRE concern, the policy becomes theatre and the next incident is met with improvisation. The right move is to apply the agreed policy now - freeze non-critical releases and direct effort to reliability - or to renegotiate the SLO openly with the business and accept the visible consequence. Quietly exempting this launch teaches everyone the budget is decorative. If the migration proceeds, it needs its own risk treatment: staged rollout, canary, tested rollback, and a plan for what happens if the remaining budget is exceeded.
2. The SRE team spends 65% of its time on toil. What should change, and how does that relate to the error budget?
Read the worked answer
High toil is a leading indicator of future budget burn: manual deployment and repetitive alert triage create both delay and error. Treat toil percentage as a first-class metric with a target, commonly at or below 50%, and spend error-budget recovery effort on the largest sources - automate deployment steps, reduce noisy alerts, self-service access requests. Use blameless post-incident reviews to find the system conditions that made manual steps necessary, not the person who ran them. Invest in canary releases, automated rollback and better observability before adding scope. If the business genuinely needs the feature now, it must fund reliability capacity alongside it; otherwise the same conflict returns next month with a worse service. The point is to make reliability work visible and funded, not to win an argument between two teams.