Halden Insurance: spending the last of an error budget

The decision belongs to the product owner, but not unilaterally and not on commercial value alone. A pre-agreed error budget policy exists precisely to remove this argument from the heat of the moment; if Halden has one, apply it. If it does not, that absence is the real finding.

Substance: 94% consumed with five weeks left means roughly one more bad week exhausts the budget. The honest options are not "ship" or "don't ship":

• Ship behind a feature flag with a progressive rollout and an automatic rollback threshold, so exposure is bounded and reversible.
• Ship only after the specific failure mode is addressed - the connection-pool setting is a known, identified risk that has already caused two severity-1s. That is not general reliability work, it is a fix to a diagnosed defect, and is cheap relative to the exposure.
• Freeze feature work until the budget recovers, which is what the policy implies, and accept the £2m delay.

"Ship it and fix it later" is only defensible if "fix later" has a name, a date and a capacity allocation - otherwise it is a decision to keep borrowing.

The SRE refusal must be handled as a governance breakdown: escalate to the service owner with the SLO data, not as an engineering-versus-product fight. If leadership overrides, record the accepted risk explicitly so the next incident review starts from an informed position rather than an assumption.

An error budget is the right shape of control - it converts reliability into a currency that a product owner can trade against feature value - but this one is too coarse to be useful.

Problems: it is a single availability target for an entire service, so it cannot distinguish "the site is down" from "quotes are 200ms slower", yet they have very different customer and regulatory consequences. It also gives no sense of which work is most effective at recovering budget, so "spend a sprint on reliability" becomes an unbounded request that product owners reasonably resist.

Improvements:

• Multiple SLOs on the things that matter: successful bind completion (not just availability), latency at the 95th percentile, and correctness of the premium calculation - an insurance quote that is wrong but fast is a worse outcome than a slow one.
• Burn-rate alerting rather than a cumulative percentage, so the team sees a fast burn in hours rather than discovering consumption at the end of the quarter.
• A named policy with consequences written in advance - who is notified at 50%, 75% and 100%, and what stops.
• Tie budget recovery to a prioritised reliability backlog so the spend is targeted at the two known connection-pool defects rather than generic hardening.

Finally, review whether the 99.9% target was ever validated against customer expectation; an arbitrary target creates arbitrary arguments.

The recurrence risk sits in three places: undiagnosed repeat failures, slow patch flow, and a culture where bad news arrives late.

• Blameless post-incident review with a completion obligation. Two severity-1s from the same connection pool means the first review produced no effective action. Track action items with owners and dates, and report the percentage closed on time - that metric, not the incident count, tells you whether the learning loop works.
• Attack the patch backlog statistically. 47 patches at a median age of 31 days suggests a dependency-update path that requires manual effort or fear of breakage. Automate dependency updates, run them continuously rather than in a quarterly push, and fund a small amount of capacity permanently for it. Security work batched to the end of a quarter is what creates the conflict you are in.
• Shift the risk conversation earlier. Surface SLO burn and outstanding patches on the same dashboard as delivery progress, reviewed in the same weekly forum, so the product owner is never surprised at release time.
• Exercise the failure modes deliberately - game days or targeted fault injection on the quote-to-bind path, starting read-only and in lower environments - so recovery capability is proven rather than assumed.
• Protect the SRE function's ability to dissent. If saying no is career-limiting, the next objection will simply be silent, and you will find out from a regulator instead.