Troubleshooting Applications Using APM
Troubleshooting applications using APM
This domain tests a disciplined diagnostic workflow: scope the problem by service, environment, version, and time; compare error and latency signals; use traces and flame graphs to find the failing hop; then confirm with logs, deployment data, and infrastructure metrics. The exam favors method over guesswork.
A repeatable triage sequence
First bound the incident: filter by service, env, version, and a time window around the onset. Compare error rate and latency percentiles against a baseline period. Decide whether the symptom is errors, latency, throughput, or saturation. Open a representative failing trace, not the slowest one by default, and read the flame graph from root to leaf. Look for a downstream span that dominates duration, a repeated database span, or an error span with a stack trace. Check the service's dependencies on the Service Map. Correlate logs using the trace ID injected into log lines. Compare versions if a deployment coincides with the change. Check host and container metrics for CPU, memory, and network saturation before blaming application code.
Reading self time versus total time
A parent span's duration includes its children. If a 2,000 ms request has a 1,900 ms database child, the parent is not the bottleneck. If many small database spans appear in one request, suspect an N+1 query pattern. If a span is missing entirely across a queue or thread boundary, suspect broken context propagation rather than a fast call.
Worked example
After deploying checkout version 2.3.0, p95 latency doubles. Filter Trace Explorer to service:checkout env:prod version:2.3.0 and compare with version:2.2.0. The flame graph shows twelve sequential db spans for SELECT inventory in a single request, while 2.2.0 shows one. That is an N+1 pattern introduced in the release. Confirm by correlating logs with the trace ID, then roll back or fix the query. Also verify the Agent's trace intake is healthy so missing spans are not mistaken for a code fix.
Traps
Blaming the parent service when a child consumes the time. Assuming every missing span is an instrumentation bug when head-based sampling can drop traces. Searching outside indexed retention and concluding the trace never existed. Ignoring version and environment filters, which mixes healthy and unhealthy populations. Treating an unset error tag as proof that no error occurred.