Northgate Bank: Whose SLO Is the Anomaly Scoring Service?
Northgate Bank runs a log-anomaly scoring service: 2.5 million log lines per minute are scored and the top-scoring clusters are pushed into the on-call console used by 40 SREs. The model is trained and retrained by a data science team on its own MLOps pipeline, which pushes a new model version to the Kubernetes inference service roughly twice a week without a change ticket. SRE owns the on-call rota and holds a latency SLO on the inference service. Nobody owns the model's output quality. Last month a retrain changed the score threshold silently; scored cluster volume tripled, the on-call console became unusable for two days, and two Sev2 incidents were acknowledged late. The retrain pipeline itself has no monitoring, and feature freshness for two upstream log parsers is checked by hand.
1. Separate the reliability objects here and state who should own an SLO for each.
Read the worked answer
There are three distinct objects, and treating them as one is the root cause.
The inference service (latency, availability, throughput) is a production service. SRE owns its SLO and its pager. That part already works.
Model output quality - for example the rolling precision of top-ranked clusters against confirmed incidents, and scored-cluster volume per day - is a product-quality indicator. The data science team owns it, but it must live on the same dashboard as the service SLO and carry an explicit acceptable band. A retrain that triples volume is an outage of the AIOps function even though every pod is healthy.
The retraining pipeline has its own MLOps SLOs: last successful training run, feature freshness per upstream parser, schema validation, and evaluation-gate pass rate.
The incident was not a modelling failure but a governance gap: a change to production behaviour shipped with no review, no canary, and no comparison of alert volume before and after.
2. Is a silent model-quality regression an incident? Justify the paging policy you would put in place.
Read the worked answer
Yes, when it degrades an operational outcome SREs depend on. Define the impact in SRE language: how fast does it burn the actionability SLO?
Mapping to propose: complete silence, where no clusters are surfaced - Sev1, page immediately, because the safety net is gone and nobody knows. Volume tripling or precision collapse - Sev2, page the model owner within business hours and auto-fallback to the previous model version. Feature staleness beyond the stated freshness SLO - Sev3, ticket.
Crucially, route model-quality pages to an ML on-call rotation, not to the SRE rota. Paging SREs for something they cannot diagnose trains them to ignore the pager. SRE owns the fallback mechanism and the SLO definition; the model owner owns the fix. Automate the fallback so the human decision is only whether to accept the new model, not how to get the old one back.
3. The data science team argues that requiring shadow evaluation before each of two weekly retrains will slow delivery. How would you resolve this with error budgets rather than a blanket ban?
Read the worked answer
Reframe it as a budget negotiation. Every retrain spends risk, so give the model owner a measurable allowance and let them choose how to spend it.
Concretely: define an actionability SLO (say 95% of top-ranked clusters link to a real incident over a 28-day window) with an error budget. While budget remains, allow rapid promotion provided each release passes an automated gate on a replayed window of historical logs and cluster volume stays within a band of the prior version. When the budget is exhausted, freeze automatic promotion and require shadow mode plus a named approver.
This gives the team speed in normal conditions and forces caution precisely when the service is already degraded. It also makes the argument quantitative: if two weekly retrains are genuinely safe, they will not exhaust the budget, and the policy costs nothing. If they are not safe, the burn rate proves it without anyone having to win a policy debate.