Application Observability: Services, Traces and Code-Level Evidence
Application Observability tests whether you can take a slow or failing request and explain it from instrumented evidence: distributed traces, service and endpoint metrics, code-level detail, and topology. Expect scenarios where the top-level service looks guilty but the data proves otherwise.
Instrumentation model
- OneAgent instruments processes; a process group is promoted to a service, and requests into it become endpoints (methods, URLs, queue consumers).
- Distributed tracing is stitched across tiers with trace-context headers; asynchronous hops through Kafka, JMS or HTTP queues appear as separate traces linked by messaging metadata rather than one continuous span tree.
- Traces show per-span timing plus response-time hotspots that attribute duration to specific methods and database statements.
- Request attributes extract values from requests for filtering and segmentation; capture and filter rules decide what is recorded at all.
Rules that change your results
- Service detection rules determine which processes merge into a service. Changing them creates new entities and breaks historical comparison.
- Sensors and span capture can be disabled per technology, producing invisible blind spots that look like healthy fast code.
- OpenTelemetry spans may be ingested with or without OneAgent; SDK-side sampling limits what Davis can ever see.
- Apdex is computed from request or user-action thresholds, not from infrastructure metrics, so an Apdex drop is not automatically a host problem.
Worked example
A checkout endpoint keeps a stable p50 but p99 hits three seconds. Opening a slow trace shows the payment child span consuming 2.6 seconds while the payment service reports normal server-side time. The delay is therefore client-side waiting: connection pool exhaustion, DNS, or retries. Failure detection confirms repeated retry spans. Reasoning: subtract child span durations from the parent before accusing the parent; the parent only looks slow because it waits.
Traps that cost marks
- Blaming the entry-point service for time spent in a child or asynchronous hop.
- Ignoring clock skew between hosts when reading absolute span timestamps.
- Editing capture or service detection rules mid-incident and losing comparable history.
- Treating a disabled sensor as proof that a code path is fast.