Monitoring, Observability, and Incident Response
Monitoring, Observability, and Incident Response
This domain tests how you instrument IBM Cloud workloads, turn signals into actionable alerts, and run incidents with clear roles and evidence.
Core rules
- Use the four golden signals: latency, traffic, errors, and saturation. Add dependency and queue depth where relevant.
- IBM Cloud Monitoring (Sysdig) collects metrics; IBM Log Analysis aggregates logs; IBM Cloud Activity Tracker records account-level API activity; VPC Flow Logs show network flows. Correlate them by time and resource ID.
- Alert on symptoms that affect SLOs, not on every CPU spike. Use multi-window burn-rate alerts and route by severity.
- Observability needs high-cardinality context: service, version, zone, request ID, and trace ID. Distributed tracing exposes latency across service boundaries.
- Incident response uses an incident commander, a scribe, a communications lead, and a technical lead. Runbooks shorten MTTD and MTTR.
Worked example
A p99 latency alert fires for a service in IBM Cloud Kubernetes Service. The dashboard shows CPU is normal. Logs show timeouts to a database; traces show the delay in one dependent call. The incident commander declares a SEV-2, routes traffic away from the affected zone, and opens a postmortem. The fix is a connection-pool limit, not a node restart. Reasoning: the symptom is latency, but the cause is downstream saturation.
Traps
Dashboard-only monitoring without alerting on user-facing SLOs fails. Static thresholds create alert fatigue. Missing request IDs makes correlation impossible. Declaring an incident without a commander creates conflicting actions. Do not confuse Activity Tracker audit events with application logs.