Indexer Capacity and Tiered Retention
Your LogRhythm deployment has one Data Indexer with 18 TB usable capacity and ingests about 350 GB per day. All log sources currently have 90-day index retention. The indexer is 84% full, and 14-day searches intermittently time out during business hours. A new compliance requirement says firewall and authentication logs must be retained for 365 days, while proxy and DNS debug logs need only 30 days. Budget is approved for a second Data Indexer, but procurement and installation will take six weeks. In the meantime, you must lower the risk of running out of index space without destroying evidence for an open investigation that covers the last 45 days. You also need a final design that meets the 365-day requirement without making all searches slower.
1. What immediate retention and indexing changes would you make, and how would you avoid deleting data needed by the open investigation?
Read the worked answer
I would classify log sources by volume, investigative value, and compliance requirement before changing any policy. Proxy and DNS debug logs are the first candidates to move from 90-day index retention to 30 days, because they are high volume and low compliance value. Before applying that change, I would confirm the open investigation does not depend on those sources for the affected hosts and time range, and I would export or archive any in-scope records. I would not reduce firewall or authentication retention yet, because those are the sources now required for 365 days and may contain evidence. I would apply retention per log source type, not globally, and monitor the purge. For firewall and authentication, I would keep the current 90-day index and plan archive-based long-term retention until the second indexer is available, rather than extending 365 days on an already-full indexer. The immediate goal is to create headroom without irreversible loss.
2. How would you design retention and search performance after the second indexer arrives, and which metrics prove it is working?
Read the worked answer
I would use tiered retention: a shorter hot-index window for high-volume sources, a longer indexed window for compliance-critical sources, and archive storage for the full 365 days where search latency is acceptable. Firewall and authentication might keep 90 to 180 days indexed and the remainder archived; proxy and DNS stay at 30 days indexed. I would distribute index load across both Data Indexers and confirm search behavior across the distributed index rather than assuming the second node only adds disk. I would set alerts on disk utilization at 70% and 80%, daily index growth, search latency at the 95th percentile, index queue depth, EPS, Data Processor CPU, and dropped-event counters. Success means searches over the required window meet the service level, disk growth has predictable headroom, and no compliance source is purged before its archive copy is verified.