Ingestion Volume, Unparsed Sources, and PCI Retention
Metro Retail's New-Scale tenant ingests 9,500 EPS from 410 log sources. Zeek connection logs are 38% of volume, DNS 22%, and firewall 15%. Finance says the monthly data budget was exceeded in week three. The ingestion dashboard shows 47 sources unparsed and duplicate firewall events because two syslog relays forward the same stream. Hot searchable retention is 30 days, but PCI requires one year for cardholder-adjacent logs. The CISO also wants to add 60 new cloud sources next quarter. Two analysts complain that searches over 14 days time out. The platform team has one parser developer and no staging tenant; all parser changes go straight to production. The company has a change advisory board on Thursdays.
1. In what order would you diagnose and reduce volume while preserving detection coverage? What would you explicitly not drop?
Read the worked answer
Order matters: stop paying for duplicate and low-value bytes before tuning parsers. First remove the double relay path for firewall logs by making one relay authoritative or by deduplicating on a stable event hash at the relay; confirm the drop with source-side and platform counts. Second, quantify Zeek connection and DNS value by detection use case. Keep DNS for tunneling and beaconing but consider filtering known-benign high-volume internal resolvers; for Zeek connection logs, retain external and uncommon service traffic and aggregate or drop fully internal chatty flows only after detection owners sign off. Third, fix the 47 unparsed sources because unparsed events still consume storage and create blind spots; map to existing parsers or queue custom parsers. Use collector-side filters for verbose fields, not whole events, where possible. Do not drop authentication, endpoint, or cardholder-adjacent logs. Re-measure EPS and cost per source after each step and document detection coverage before and after.
2. How would you design retention and search performance to satisfy PCI while keeping investigations usable? What evidence would an auditor need?
Read the worked answer
Tier retention by data class. Keep 30 days hot for investigation, move cardholder-adjacent logs to cold or searchable archive for 365 days, and verify that searches over the archive meet the analyst SLA. If 14-day searches time out, the issue may be query scope and concurrency rather than retention tier; restrict fields, use indexed fields, and schedule heavy searches. For PCI, prove that cardholder-adjacent sources are ingested, parsed, time-synchronized, access-controlled, and retained for one year. Outcomes Navigator can show coverage and gaps, but auditors will want evidence: source inventory, retention configuration, access review, and sample searches. Document any filtered events and the risk acceptance. Adding 60 cloud sources next quarter should be capacity-planned now, because retention changes and new sources compete for the same budget.
3. With no staging tenant, how would you manage parser changes safely?
Read the worked answer
With no staging tenant, create a controlled production validation lane. Clone a small set of representative raw events, develop the parser against those samples offline, and test in a non-production project if the vendor offers one; if not, use a dedicated low-volume log source and a canary source group. Version every parser, record the sample event IDs, expected normalized fields, and a rollback copy. Deploy only during the Thursday change advisory board with a named validator, then compare parsed versus unparsed counts, key field population, and detection output for 24 hours. Never edit a parser used by a critical detection without a rollback. Ask the vendor for parser review or open a support case if the format is nonstandard. Track parser success rate as a service metric and add a second parser developer through training, because one person is a single point of failure.