Hybrid Cutover with Site Collectors, Proxy Resets, and Clock Skew
Northwind Health runs LogRhythm 7.5 on-premises with three Data Processors, two Mediators, and four Indexers, peaking at 18,000 EPS and retaining 120 days. It is adding Exabeam New-Scale for cloud analytics and plans a 90-day parallel run. Six Site Collectors will sit in two data centers; the secondary site egresses through an inspecting proxy that resets long-lived TLS sessions every 30 minutes. Domain controllers show a seven-minute clock skew after a virtualization migration, and several network devices still send UDP/514 syslog that drops bursts above 4,000 EPS. The cutover is four weeks away. The CISO wants no blind spots during the transition and a documented rollback. The network team can provide 200 Mbps sustained egress per data center, but change windows are limited to two hours on Saturdays. The security team has two engineers who can test parsers and one analyst who can validate detections.
1. What prerequisites and tests would you complete before cutover to prove Site Collector throughput, resilience, and proxy compatibility? Give the metrics and pass/fail thresholds you would use.
Read the worked answer
Start with a per-site capacity and path test, not a parser test. Calculate peak EPS per collector from the 18,000 EPS global peak and the source-to-collector mapping; keep each collector below 60-70% of tested sustained throughput so bursts and retries fit. For the proxy site, test TLS 443 through the inspecting proxy with a long-running synthetic source for at least 90 minutes to catch the 30-minute reset; if resets break the connection, request a bypass for the collector destination or use a supported proxy configuration and confirm buffering and spooling behavior. Run a 24-hour replay of representative sources and compare source-side counts with New-Scale ingestion counts by parser and minute. Pass criteria: less than 1% unexplained loss, no parser queue backlog, collector CPU below 70%, memory below 80%, and no gaps over two minutes. For UDP/514 devices, move to TCP/TLS or a relay that buffers; do not accept burst loss. Document throughput results and the proxy exception in the cutover plan.
2. The seven-minute clock skew is unresolved. Explain how it affects New-Scale sessionization, timelines, and correlation, and what you would fix first.
Read the worked answer
New-Scale sessionization links events into user sessions using time-ordered activity. A seven-minute skew across domain controllers means logons, privilege changes, and endpoint events can be placed in the wrong order or outside the session window. Timelines will show a logon after the resource access it caused, and correlation rules with short windows can miss or mis-attribute activity. Fix the source first: enforce NTP on hypervisors and domain controllers, verify with w32tm or chrony, and confirm the offset is under a few seconds before cutover. On the platform side, verify all sources and collectors use UTC and that parsers preserve the original event time rather than ingest time. Do not mask the issue by widening correlation windows; that increases false positives and hides the root cause. Re-baseline UEBA after the fix because skew-distorted sessions can pollute peer-group models.
3. Design the cutover and rollback sequence by log source group. What evidence would make you pause or reverse?
Read the worked answer
Cut over by log source group, not by data center. Start with low-risk, high-volume sources to prove throughput, then identity and endpoint, then network and cloud. For each group: freeze parser changes, capture a 24-hour baseline in LogRhythm, enable the New-Scale source, run parallel for 72 hours, and compare event counts, parser success, and detection output. Pause if unexplained loss exceeds 1%, if a critical detection has no New-Scale equivalent, or if the proxy site shows reconnects. Rollback is per group: disable the collector input, keep LogRhythm forwarding unchanged, and revert DNS or relay changes. Keep LogRhythm retention intact for the full 90 days, not just until the parallel run ends. Schedule the two-hour Saturday window for relay changes and have a named engineer on each side with a go/no-go checklist.