Logging That Drops 15% and Fills a 64 GB Partition in 36 Hours

Math: 8,000 events/s x 300 bytes = 2.4 MB/s = about 207 GB/day. Over 90 days that is roughly 18.7 TB; over one year, about 75 TB (before indexing overhead, which typically adds 20-50%). Peak 25,000 events/s x 300 bytes = 7.5 MB/s, or about 648 GB/day if sustained.

Conclusion: 2 TB cannot hold 90 days of full session logs at this rate - it holds roughly 9-10 days. The requirement as written is not achievable with the current design. Note also that bandwidth is not the bottleneck: 2.4 MB/s is trivial for 1 GbE (about 2% utilization). The drops are buffer, disk-write, or CPU related, not link capacity.

Options, in order of preference:

• Reduce volume at the source: session-end logs for every allow dominate the stream. Replace with summarized traffic/flow records for allowed traffic and keep full event logs for denies, threats, policy changes, VPN and authentication events. This is usually a 5-10x reduction.
• Compress and tier: raw compressed logs (3-5x typical) for long retention, indexed/hot storage for 90 days only. Recalculate: 18.7 TB / 4 ≈ 4.7 TB compressed - still above 2 TB, so reduction at the source is required, not optional.
• Aggregate/roll up: keep 90 days of detailed security events, and daily summaries plus flow aggregates for the remaining retention.
• Add storage or a second collector if the business genuinely needs full session detail for a year; state the cost and get a decision in writing rather than silently under-retaining.

Whatever is chosen, document the actual retention achieved per log type, because 'one year' as an undifferentiated requirement will not be met by any affordable design at 8,000 EPS.

Move off fire-and-forget UDP to a reliable, authenticated transport: syslog over TLS (TCP 6514) with per-device certificates, or an agent-based forwarder with disk-backed buffering. Architecture changes that matter more than the protocol:

• Buffer locally on each firewall and forward with acknowledgement, so a collector outage or burst does not lose data. Size the local buffer for at least 24-48 hours at peak rate: at 25,000 EPS x 300 bytes, that is ~648 GB/day, which will not fit in 64 GB - so either rate-limit what is buffered (security events only) or accept that session logs are best-effort and security events are guaranteed.
• Use at least two collectors behind a load balancer or per-device fan-out, so a single disk stall does not drop everything.
• Apply per-source rate limits and queue depth monitoring; alert on queue saturation and on collector ingest rate dropping below the expected baseline.
• Separate log classes: security/critical events get reliable delivery and long retention; high-volume session logs can be sampled or summarized.

Tradeoffs to state: TCP/TLS introduces head-of-line blocking and backpressure - a slow collector can stall the sender, which is why local buffering and rate limits are mandatory. Certificates add operational overhead (expiry, rotation). TLS does not by itself guarantee delivery if the sender drops on queue-full; verify the buffering behavior, not just the port number.

Time problem: only one device has NTP, and it is 4 minutes fast. A 4-minute skew across three firewalls plus the collector makes cross-device correlation unreliable: an attack seen on two firewalls appears as two unrelated events, and a session's start and end can appear out of order relative to the collector's receive time. Fix: authenticated NTP (or PTP where available) on every firewall and the collector, all in UTC, with an alert if any source drifts more than a few seconds. Store both event time (from the device) and receive time (at the collector), and do not silently trust device time. Validate after the change with a known event seen on all three devices.

Prioritization if volume must be cut, highest value first:

1. Denies and drops, especially from untrusted zones; policy changes and admin logins; VPN/authentication successes and failures; threat/prevention events; NAT and routing changes. These are the events investigations actually need.
2. Session starts for allowed traffic to sensitive destinations, or sampled allows.
3. Session-end records for allowed, internal-to-internet traffic - the highest-volume, lowest-value class. Summarize or drop these first.

Also suppress known-noise sources (internet scanners hitting the implicit deny) with rate-limited summary logs rather than per-packet entries, and log the suppression itself so the gap is documented. The goal is that a reviewer can reconstruct who did what to which asset and when - not that every packet is retained.