OSMR - Operational Systems Monitoring and Response
Domain overview
This domain tests how well you can design, operate, and troubleshoot monitoring and incident response for production systems. Questions are scenario-based: you are given a service, an SLO, and a set of symptoms, then asked what to alert on, how to interpret the error budget, or how to run the incident. Tool names are less important than the reasoning behind service level objectives, alert thresholds, severity, and post-incident review.
Core rules and concepts
- An SLI is a measured ratio, such as successful requests divided by total requests. An SLO is a target for that SLI over a window. An SLA is a contractual consequence, not an internal operating target.
- Error budget = 1 − SLO. A 99.9% SLO over 30 days allows 0.1% of requests to fail, which is about 43.2 minutes of full outage if translated to time.
- Alert on user-facing symptoms - latency, error rate, saturation - not on every possible cause. Multi-window burn-rate alerts catch both fast and slow budget consumption.
- Severity levels drive escalation and communication. Define SEV1–SEV4 with explicit entry criteria and assign one incident commander.
Worked example
A service has a 30-day SLO of 99.9% availability measured per request. In the first 10 days it serves 2,000,000 requests and 3,500 fail. The full-window budget is 0.1% of the expected 6,000,000 requests = 6,000 failures. Actual burn is 3,500 / 6,000 = 58% in one-third of the window. The observed error rate is 3,500 / 2,000,000 = 0.175%, so the burn rate is 0.175% / 0.1% = 1.75×. At that rate the budget exhausts before day 30, so a slow-burn ticket is justified even though no single hour looks catastrophic.
Traps that cost marks
- Confusing the SLI with the SLO: the SLI is the measurement, the SLO is the target.
- Treating the error budget as a daily allowance instead of a rolling-window budget.
- Paging on high CPU alone; CPU is a cause, not a user symptom.
- Declaring an incident resolved before the customer-facing symptom clears and before postmortem action items have owners and dates.