LTF12 study materials
LTF12 study resources

An SLO That Breaches Before the Budget Cycle Does

LTF12 · Put your knowledge to work in context.

All case studies

An SLO That Breaches Before the Budget Cycle Does

A nightly ETL pipeline must deliver by 06:00. It currently finishes at 05:40, having started at 01:00. Input volume is growing about 4% per week and runtime has tracked volume almost linearly. Nine weeks remain in the quarter end-to-end, but reserved compute expires in five weeks and a budget freeze blocks new spend until next quarter. Three levers exist: buy four additional workers (30% faster, requires an exception approval that averages three weeks), optimize the worst query (two engineers for three weeks, estimated 20–40% gain, uncertain), or shorten raw-data retention (compliance review required, no estimate available). The business has not been told the deadline is at risk.

1. Model when the SLO actually breaks. Show the arithmetic rather than asserting a date.

Read the worked answer

Baseline: 01:00 to 05:40 is 280 minutes of runtime against a 300-minute deadline, so headroom is 20 minutes.

Headroom as a fraction: 20 / 280 = 7.1% of runtime may be added before breach.

Compounding growth: (1.04)^n = 1.0714, so n = ln(1.0714) / ln(1.04) ≈ 0.0690 / 0.0392 ≈ 1.76 weeks.

Conclusion: the pipeline breaches in roughly 12 days - not at the end of the quarter, and not when the reserved compute expires in five weeks. Both of those frames are dangerously late.

Caveats that matter for the decision:

  • The linear assumption is generous. If the growth is in small files rather than bytes, per-record overhead makes runtime grow faster than volume, and the breach arrives sooner.
  • Sub-linearity is possible if fixed startup cost dominates, but at three-plus hours of runtime that is unlikely.
  • A "finish by 06:00" SLO usually has downstream consumers; the real breach may bite earlier than the pipeline's own timestamp.

The decisive output: the arithmetic moves the problem from a capacity conversation in five weeks to an incident-risk conversation now. That reframing is what unlocks an approval that a growth chart would not.

2. Which lever do you pull first, given that no single lever is both fast and certain?

Read the worked answer

Start the query optimization today and file the exception approval the same day. Do not wait to see whether the optimization works.

Reasoning per lever:

  • Query optimization (2 engineers, 3 weeks, 20–40%): the only lever you control without a decision from someone else. Its low end, 20%, is nearly triple the 7.1% headroom, so even a pessimistic result buys roughly four to five more weeks. It is also permanent, unlike purchased capacity.
  • Four workers (30% faster, 3-week approval): the backup. Filing the exception now means it can land around week 4 - after the breach date but before the compute reservation expires. Filing it after the optimization proves insufficient pushes it to week 6 or later, past both events.
  • Retention reduction: the least predictable lead time and the only lever that destroys data. Hold it in reserve; spending an irreversible option before you need it is bad sequencing.

Why parallel, not sequential: the levers have independent lead times and one is not contingent on the other's outcome. Optimizing first and buying only if needed is the intuitively careful path, and it is the one that misses the deadline.

Condition that changes this: if the first week of the optimization shows no measurable gain, escalate to the retention review immediately rather than waiting for week three.

3. Write the escalation ask. What exactly are you requesting, with what numbers attached?

Read the worked answer

The ask: pre-approve the compute exception now, contingent on the optimization underdelivering, and release two engineers from the current roadmap for three weeks.

The numbers to attach:

  • Breach in approximately 12 days on current trajectory; headroom is 20 of 300 minutes.
  • A 4% weekly growth rate; every week of delay consumes roughly half the remaining headroom.
  • Optimization covers 20–40%, yielding four to five additional weeks at the pessimistic end.
  • Additional workers cover 30%, and the approval averages three weeks - which is why it must start before the outcome is known.
  • Reserved capacity expires in five weeks, independent of the SLO.

Framing that gets a decision: present it as an insurance premium against a dated event, not as a capacity request. State explicitly what happens if declined: a missed 06:00 delivery within two weeks, with downstream consumers affected, and a retrofit under time pressure that is more expensive than either lever.

Also state what you are not asking for: no retention change today. Naming the option you have deliberately withheld makes the request credible and shows the escalation is scoped, not a panic sweep.

What to omit: speculation about who caused the growth. It is irrelevant to the decision and turns a capacity question into a blame question.