I85PREP
⌘ K
Observability Foundation study materials
Observability Foundation study resources

Observability Foundation: Full-Stack Observability

Observability Foundation · Work through your exam, one topic at a time.

All guide sections

Full-Stack Observability

Full-stack observability

This domain tests whether you can reason across every tier a request touches - browser or device, edge, network, service, datastore - and the business layer above them, and whether you can correlate those tiers into one narrative.

The layers that must be covered

  • Client and RUM - page load and Core Web Vitals, JavaScript errors, mobile crashes, session replay.
  • Edge and CDN - cache hit ratio, origin offload, TLS negotiation, DNS resolution.
  • Application - services, serverless functions, queues, scheduled jobs.
  • Data - query latency, connection pool saturation, cache hit rate.
  • Business - orders per minute, conversion, feature-flag cohorts.

Correlation is the whole point

Inject the same trace context from the browser via fetch/XHR headers so a RUM session links to the backend trace. Propagate baggage for tenant or plan tier. Keep resource attributes consistent across tiers, maintain one service map, and synchronise clocks - cross-signal joins silently mislead when timestamps disagree.

Methods

Golden signals (latency, traffic, errors, saturation) per service; RED for request-driven services; USE for resources. Measure SLIs as close to the user as possible: a server-side 200 ms response means nothing if the page took six seconds to render.

Worked example

Support reports slowness only for mobile users in one region. Server p99 is unchanged. RUM shows LCP of 6.2 s versus 2.5 s elsewhere; CDN cache-hit ratio in that region fell to 40% after a deploy changed cache keys, so the origin saw three times the requests. Server spans show the extra load as queue time, not service time, which is why backend metrics looked clean. Reasoning: only the client and edge tiers could see this failure, and only trace correlation proved the link to the deploy.

Traps

  • Measuring only the server and calling it full-stack.
  • Sampling away errors at the edge or in the browser.
  • RUM deployed without trace-context injection, so sessions cannot be joined to traces.
  • Ignoring clock skew between client and server.
  • Assuming CDN or DNS behaviour is visible in application telemetry.

DevSecOps and AIOps Impact on Observability

DevSecOps and AIOps impact on observability

This domain tests how security automation and machine-learning operations change what you collect, how you alert, and what you can trust.

DevSecOps changes the signals

Security telemetry becomes a first-class signal rather than a separate SIEM-only concern: authentication failures, WAF and rate-limit events, runtime detection from eBPF, audit logs, and supply-chain evidence such as SBOMs and build provenance. Pipeline telemetry matters too - DORA metrics (lead time, deployment frequency, change failure rate, time to restore) come from CI/CD events, and deployment markers annotated onto dashboards explain most sudden metric shifts. Compliance evidence is a by-product: retention, immutability and least-privilege access to telemetry must be designed, and PII must be redacted at the Collector before export.

AIOps changes the consumption

AIOps adds anomaly detection on seasonal metrics, alert correlation and deduplication, log clustering, root-cause suggestions, and LLM assistants that query telemetry in natural language. All of it depends on labelled, well-baselined data and a feedback loop. Keep deterministic SLO-based alerting as the backbone; AIOps sits on top as an accelerator, not as the paging system.

Worked example

A database failover produces 300 alerts in ten minutes across services. A correlation engine groups them into one incident using service topology and temporal proximity, and the on-call sees a single item with 300 linked signals. The trap is accepting the suggested root cause without checking a representative trace, because the model learned the old topology. Reasoning: correlation reduces noise, but it does not establish causation.

Traps that cost marks

  • Shipping security logs to a separate silo with no shared resource attributes.
  • Putting PII or credentials into span attributes and logs.
  • Enabling AIOps before SLIs and SLOs exist, so there is no ground truth.
  • Trusting anomaly output after an architecture change without re-baselining.
  • Treating pipeline metrics as out of scope for observability.

Organizational Observability Maturity

Organizational observability maturity

This domain tests whether you can assess and advance an organisation's observability practice, and whether you know that maturity is a practice ladder, not a purchasing ladder.

A practical level model

  • Reactive - host and CPU dashboards, threshold alerts, no tracing; teams learn about incidents from users.
  • Defined - centralised logs and metrics, some tracing, runbooks, on-call rotations.
  • Proactive - SLIs and SLOs with error budgets, blameless postmortems, OpenTelemetry as the instrumentation standard.
  • Observed - observability-driven development: new features ship with instrumentation, engineers answer novel questions with high-cardinality queries, telemetry cost is governed.
  • Optimised - continuous verification, assisted correlation, business KPIs wired to SLOs.

Dimensions worth scoring

  • Coverage of critical user journeys, not percentage of hosts instrumented.
  • Time to detect and time to understand - MTTR alone is misleading, since fast recovery can mask poor detection.
  • Ownership: do teams instrument their own services, or is there a central bottleneck?
  • Standards adoption: semantic conventions, shared resource attributes, consistent trace propagation.
  • Cost governance: retention tiers, sampling policy, spend per service.

Worked example

An organisation has 100% host metrics, traces on 60% of services, no SLOs, and postmortems that name an individual. The highest-leverage next step is not more tracing. It is defining SLOs for the five top user journeys and making postmortems blameless, because without them the extra spans will not be consulted and incidents will keep being hidden. Reasoning: maturity gains come from the practice that unblocks the next one, not from adding the next tool.

Traps

  • Scoring maturity by tool count, licence tier or dashboard quantity.
  • Using MTTR as the single health metric.
  • Claiming a high level because a vendor was purchased.
  • Measuring instrumentation by hosts rather than by user journeys.
  • Skipping SLOs and jumping straight to AIOps or predictive tooling.

Network and Container-Level Observability

Network and container-level observability

This domain tests whether you can see failures that live below the application: network paths, DNS, service mesh behaviour, and the ephemeral, high-cardinality world of containers and Kubernetes.

Network layer

  • Flow data - VPC flow logs, NetFlow/IPFIX, eBPF-derived flows; good for who-talked-to-whom, retransmits and RSTs.
  • DNS - resolution latency and NXDOMAIN rates; a leading cause of "the app is slow" that application traces often miss.
  • Service mesh - per-route latency, status codes, mTLS and retry counts. Retries inflate request counts and can duplicate spans; read them separately.
  • Connection health - TLS handshake failures, pool exhaustion, packet loss, saturation at the load balancer.

Container and Kubernetes layer

Pods are ephemeral, so identity must come from resource attributes: k8s.namespace.name, k8s.pod.name, k8s.deployment.name, container.id, plus node name. Metrics sources are distinct - cAdvisor/kubelet for container resource usage, kube-state-metrics for object state such as desired versus ready replicas and restart counts, metrics-server for autoscaling only.

Collector patterns: a DaemonSet agent per node for metrics and log tailing, sidecars for per-pod isolation, and a gateway for tail sampling. Pod names and container IDs are effectively unbounded - keep them as span and log attributes, never as metric labels.

Worked example

Pods in one deployment are OOMKilled roughly every four hours. Average pod memory looks fine; kube-state-metrics shows a rising restart count; traces reveal memory growth on one request path that loads a large object per call. Reasoning: container metrics show the symptom, Kubernetes state confirms the pattern, and traces identify the triggering code path - no single source is sufficient.

Traps

  • Losing logs when a pod dies; use a node-level agent, not only sidecars.
  • Missing Kubernetes attributes, so traces cannot be mapped to a workload.
  • Prometheus label-cardinality explosions from pod hashes or container IDs.
  • Treating metrics-server as a general monitoring source.
  • Ignoring retries and thereby double-counting mesh traffic.