Tixly: An On-Sale Meltdown, an SLA Calculation and Autoscaling That Cannot Work

Arithmetic. 99.9% availability permits 0.1% unavailability. In a 30-day month that is 43,200 minutes × 0.001 = 43.2 minutes. In a 31-day month it is 44.6 minutes; in February, 40.3 minutes.

So a single 43-minute incident in a 30-day month sits right at the edge of the allowance - technically inside it, but consuming 99.5% of the month's entire error budget. Any other incident in the same month, however brief, pushes the service below the commitment.

But the arithmetic is not the answer on its own. Before saying 'no breach', check:

1. What the SLA measures. If availability is defined against the homepage or the API endpoint, a ping that returns 200 while checkout fails may not count as downtime at all - which means the customer-visible failure is invisible to the metric, and the CTO has a measurement problem worse than the outage.
2. The measurement point and method. Externally measured synthetic checks, internally measured error rates and provider-reported instance health give three different numbers.
3. Exclusions. Scheduled maintenance windows and partner-caused traffic are commonly excluded; read the definition rather than assuming.
4. The remedy mechanism. Credits are typically claim-based with a deadline; they are a contractual remedy, not a finding of breach, and their absence does not by itself prove compliance.

Answer to the CTO: probably not a breach of the letter of the SLA, almost certainly a breach of what the SLA was meant to protect, and the first deliverable is a corrected measurement definition agreed with the venues.

Four design choices combined to make the policy structurally incapable of responding.

1. The signal is lagging and wrong. CPU rises after the queue has already formed. In a traffic spike of this shape - 40,000 users in four minutes - the binding constraint was the database connection pool, not CPU. Adding application instances while the pool is exhausted makes things worse: each new instance opens its own connections against a hard limit of 200, accelerating the failure.

2. The reaction time exceeds the event. Five minutes above threshold, then a scaling action, then a ten-minute cooldown during which no further action occurs. Reacting to a four-minute spike therefore takes at least fifteen minutes, and the group scales one instance at a time rather than to a computed target.

3. The ceiling was too low and was reached. The group hit 30 instances and stayed there, so from that moment the system was saturated with no headroom and no signal to a human.

4. There was no pre-scaling for a known event. A partner's on-sale is a scheduled, predictable load event and should have been capacity-planned, not left to reactive scaling.

Changes, in priority order:

• Introduce a connection pooler/proxy in front of the database and size per-instance pool limits against total capacity, so instance count and database load are decoupled.
• Scale on the signal that predicts the pain - request rate, queue depth or latency target - using target-tracking rather than step policies, and remove the long cooldown in favour of scale-out-fast, scale-in-slow.
• Raise the maximum and pre-warm capacity ahead of announced on-sales.
• Add caching and load shedding (queue or politely reject) so the system degrades instead of collapsing.

The failure was not a lack of data; it was that nothing measured what customers actually experienced, and no process existed to act on it.

Monitoring changes:

• Synthetic user journeys, not ping. A scripted checkout path - select seats, add to basket, pay - measured every minute from outside the region. The homepage returned 200 throughout this incident while checkouts failed.
• SLO-based alerting. Define objectives from user-facing indicators (successful checkout rate, p95 latency), alert on burn rate against the error budget, and route to a human with an owner - not to a dashboard.
• Dependency metrics. Database connection utilisation, pool wait time, queue depth and saturation signals, because those were the true constraint.
• Capacity and event awareness. A calendar of on-sales and partner campaigns that feeds pre-scaling and on-call staffing.

Operational changes:

• A defined incident process: severity classification, a named incident commander, and a status page for venues so they are not discovering failures from customers.
• A blameless post-incident review within five working days, with owned actions and dates.
• A tested rollback and a load-shedding runbook.
• Regular game days and load tests that reproduce an on-sale shape before the next one.

Why not just more dashboards. Dashboards are pull-based; they only inform someone already looking. The engineer in this scenario learned about the incident from a complaint, which means the detection path was human and social rather than systematic. Adding panels to a screen nobody is watching changes nothing. The measurable target is time to detect and time to restore - push-based, owned alerts plus a rehearsed process reduce both; more graphs reduce neither.