Cutting log volume while preserving incident evidence
A fintech has 90-day retention for audit logs and 30-day retention for application logs. Daily ingest is 1.4 TB: 55% checkout debug, 20% ingress health checks, 10% audit, and 15% other. Log parsing rules fail on nested JSON from checkout because the message field is a JSON string; multiline Java stack traces split into separate lines. The team forwards logs from Kubernetes with Fluent Bit and uses New Relic Logs. A recent incident took 40 minutes to find connection-pool exhaustion because the relevant lines were debug and the parser did not extract pool.active or pool.max. The CFO wants a 40% ingest reduction in 30 days; security insists audit logs remain complete and queryable for 90 days. The observability team must cut volume without losing incident evidence.
1. Rank the first three actions to reach 40% reduction. Justify why each is safe or risky and what you would not drop.
Read the worked answer
First, drop ingress health-check logs at the collector or with a scoped drop filter. They are 20% of volume and have little incident value beyond availability, which synthetics can cover. Second, reduce checkout debug with a targeted rule: keep warnings, errors, and debug lines containing pool, timeout, or retry context, and sample the rest. That can remove roughly 20% of total volume, reaching the 40% goal, but it is the risky action because debug lines were the evidence in the last incident. Validate the rule against a representative incident query before enabling it. Third, parse the nested checkout JSON and create log-to-metrics for pool.active and pool.max; that does not reduce volume immediately, but it makes the remaining debug lines searchable and supports cheaper alerting. Do not drop audit logs, and do not apply a global debug drop filter. Use live tail and NRQL counts before and after each change to confirm the reduction and confirm that error and audit logs still arrive.
2. How would you fix parsing for the nested JSON and multiline stack traces, and how would you prove the incident is now diagnosable?
Read the worked answer
Fix multiline at the source: configure Fluent Bit's multiline parser so Java stack traces are forwarded as one record before they reach New Relic. New Relic parsing rules cannot reliably rejoin lines that were already split. For nested JSON, add a parsing rule that extracts the message field as JSON and promotes pool.active, pool.max, pool.wait, and request identifiers into attributes. Do not rely on free-text search alone. Then build a log-to-metrics metric for pool utilization and a saved query or dashboard for checkout pool exhaustion. Prove it with a game day: generate a canary connection-pool exhaustion, measure time from first symptom to identifying the pool attributes, and compare with the 40-minute baseline. If the parse fails on escaped or truncated JSON, add a fallback Grok pattern and test it against real samples. Keep the parsing rule versioned and monitor parse-failure counts so a schema change does not silently break incident response.
3. What retention and access-control choices would you make to satisfy security and cost?
Read the worked answer
Separate audit from application logs by policy, not just by query. Keep audit logs complete, restrict access through role-based permissions, and confirm the contracted retention meets 90 days; if New Relic Logs retention in your agreement is shorter, forward an immutable copy to long-term storage and keep a query path. Apply the shorter 30-day retention to application logs and consider 14 days for high-volume debug if the incident review allows it. Scope drop filters so they cannot match audit events, and require review before any filter touches security-relevant fields. Use separate data partitions or accounts if your contract supports them, so cost controls and access boundaries are explicit. Audit access should be limited to security and compliance roles; application engineers should query application logs without seeing restricted audit fields. Document the retention map and test it with a sample audit event and a sample debug event, confirming both the retention and the permission boundary.