Log Volume, Parse Failures, and Broken Logs-in-Context
Your organization ingests about 4.8 TB of logs per month across 60 services. A cost review shows 62% of volume comes from the inventory service, which has been running at DEBUG since a March troubleshooting session nobody closed out. Log parsing rules fail on roughly 14% of Java log lines because stack traces arrive as multi-line events; those failures appear in the parse-failure metric but no one has looked at them. The Go services emit single-line JSON with an RFC 3339 timestamp in a custom field named ts, and engineers complain that logs-in-context from the Go APM entities opens an empty view, while Java services work correctly. Next week a payment queue backlog incident is expected to recur, and the team wants to find its signature quickly instead of scrolling raw lines.
1. Design a plan to cut log volume without losing incident signal, and explain the order of operations.
Read the worked answer
With 62% of volume coming from one service at DEBUG, the largest cost is a configuration mistake, not a capacity requirement. Reduce in this order and measure after each step.
- Turn DEBUG off in the inventory service. Before doing it, confirm nothing depends on it: query which message patterns exist only at DEBUG, and ask the owning team whether any dashboard, alert, or runbook query reads them. If none do, this removes most of the volume at near-zero analytical cost.
- Quantify before dropping anything. Facet counts by log level and service, then find top message patterns. Look for high-volume, zero-information messages: health checks, duplicated access logs from the load balancer, retry chatter.
- Apply narrow drop filters for identified noise rather than broad level filters. Drop filters remove data at ingest and are irreversible, so each one should be named, documented, and reviewed.
- Convert to metrics where you need the signal but not the text: error counts by service, or latency percentiles derived from access lines, then drop the raw lines. Never drop or sample error-level logs.
- Only after all of the above, consider sampling for genuinely high-volume informational streams.
Track the parse-failure rate alongside volume, because unparsed lines are still billed and still unsearchable. A fix to parsing often beats another drop filter.
2. Two data-quality defects: multi-line Java stack traces failing to parse, and Go JSON with a custom ts timestamp. Diagnose each and describe the fix and verification.
Read the worked answer
Both are configuration defects at the ingestion and parsing layer, and both are fixable without touching application code.
Multi-line Java stack traces: the default line-oriented parser treats each line as a separate event, so the header parses and the frames become unrelated messages, or the whole thing fails. Add a parsing rule for that log type that merges continuation lines, anchored on the timestamp and level of the header line and capturing the stack as a single field. Test the pattern against real samples in the parsing UI before saving; a pattern that is too permissive swallows the next event's header and creates a worse problem. Verify per log type in the parse-failure metric rather than only in aggregate.
Go JSON with ts: the agent needs a recognized timestamp attribute. Add a rule that maps ts to the event timestamp and exposes the remaining JSON keys as attributes. Confirm the payload is raw JSON and not a JSON document wrapped inside a string field, otherwise the keys will never become queryable. Check that the custom timestamp is RFC 3339 as claimed, since a misformatted value silently falls back to ingest time and breaks time-series queries.
Verify both by re-running a query that previously failed, for example faceting exception type after the rule lands. Parsing rules apply to newly ingested data, so historical events are not repaired; validate over the following hour and tell the teams.
3. Logs-in-context opens empty for Go services, and the team needs to recognize the recurring queue backlog fast. Address both.
Read the worked answer
Logs-in-context is a linkage problem, not a logging problem: the APM entity and the log events must share an identifier that the UI can resolve in both directions.
Diagnose the Go services:
- Open a Go log event and inspect for entity attributes. Java works, which proves the agent path; the Go services likely ship logs through a forwarder that never received the entity identifier from the agent.
- If your linking is name-based, check for mismatch between the APM entity name and the service attribute on the logs, including pod-name suffixes. Standardize on one attribute.
- Confirm container and namespace decoration is present so the context panel can filter to the right instance.
- Fix in the forwarder configuration rather than telling engineers to search manually, then verify by opening logs-in-context from a Go entity for a request you can trace.
For the recurring backlog:
- Use log pattern clustering over the incident window instead of keyword guessing. Patterns group variable-heavy messages into stable signatures, so a new or rapidly growing signature such as a consumer stall, prefetch timeout, or publish confirm failure stands out against the baseline.
- Compare the pattern set 30 minutes before the incident with the set during it, capture the top new signatures, and convert them into a saved query plus an alert condition.
- Attach that saved query and its expected signature list to the runbook, so the next occurrence is a two-minute check rather than an expedition.