Skipper: 3,000 Alerts a Week and a Broken Trace
Skipper runs 40 services on AKS plus Azure Functions, each with its own Application Insights resource. Telemetry is a mix of ILogger, custom events, and Serilog writing to files on the Functions. Alerts fire about 3,000 times a week and 90% are closed as noise. One real outage took 40 minutes to triage because a checkout request's trace could not be followed into the payment service over Service Bus. Ingestion cost tripled last quarter, mostly from verbose debug logs in a chatty service and from dependency events. SLOs are undefined, and the team argues about whether availability means HTTP 5xx, pod restarts, or the synthetic checkout test. Sampling is at the default 100% in production.
1. Define SLOs and redesign alerting to cut noise. What do you measure and alert on?
Read the worked answer
Define SLOs per critical user journey - checkout and login - not per pod. Availability is the percentage of requests that succeed, with an explicit allowlist for expected 4xx; latency is the percentage under a threshold. Publish both in a workbook and, critically, attach an error budget so the conversation becomes how much unreliability is affordable rather than whether to page.
Alert on symptoms, not causes: SLO burn rate over a short and a long window (for example 1 hour and 6 hours). A burn-rate alert fires rarely but meaningfully, whereas static thresholds on 40 services produce exactly the 3,000 alerts you have. Keep a small, separate set of cause alerts for failures with no user symptom yet - expiring certificates, disk pressure, queue depth.
Tradeoffs: burn-rate alerting needs a couple of weeks of warm data and can under-detect very short spikes, so pair it with a per-region synthetic checkout. Pod restarts and 5xx counts are diagnostic signals, never the SLO itself.
2. Fix cross-service correlation, including the asynchronous Service Bus hop.
Read the worked answer
Standardize on W3C Trace Context and OpenTelemetry instrumentation across all 40 services; Application Insights supports W3C distributed tracing and will stitch spans into one operation.
The Service Bus hop is the part that breaks by default. Propagate traceparent (and tracestate) as message application properties when publishing, and extract and set it as the parent context on the consumer before starting any span. Without that, the consumer starts a new trace and the checkout request visibly dies at the queue.
Also attach a business correlation ID - order ID or checkout ID - as a custom dimension on every span and event, so support can search even when a trace is incomplete. Move the Functions off Serilog-to-file onto the OpenTelemetry or App Insights exporter: file logs are invisible during an incident.
Tradeoffs: manual propagation needs a shared library and enforcement in review; protect it with a test that asserts parent-child span linkage across the queue. Configure sampling to retain all errors and all slow traces, or triage gets harder exactly when you need it.
3. Control ingestion cost with sampling and telemetry configuration without hiding errors.
Read the worked answer
Reduce volume in three layers, cheapest first.
- At source: change the chatty service's debug logging to Information in production via configuration, and drop dependency telemetry for health probes and internal DNS noise. This is free and permanent.
- Sampling: enable adaptive sampling in the Application Insights SDK (or fixed-rate for predictably high-volume services), configured to exclude exceptions and failed requests. Preserving errors while sampling successes keeps triage intact.
- Consolidate: 40 separate resources make cross-service queries painful and licensing awkward; move to workspace-based resources with a smaller number of components, and set retention per data type rather than uniformly.
Use daily caps only as a backstop - a cap silently discards data, so set it well above normal volume and alert when it is approached.
Tradeoffs: sampled counts are approximate. Percentages remain valid for SLOs, but never mix sampled and unsampled counts in one chart, and keep a separate metrics path if you need exact throughput. Review cost weekly against a budget alert rather than at quarter end.