I85PREP
⌘ K
XSIAM Engineer study materials
XSIAM Engineer study resources

Broker Placement and Egress Design for a Three-Site Healthcare Deployment

XSIAM Engineer · Put your knowledge to work in context.

All case studies

Broker Placement and Egress Design for a Three-Site Healthcare Deployment

Northwind Health is deploying Cortex XSIAM across three sites: a headquarters with 12,000 endpoints and two clinics with 2,500 each. Regulatory rules prohibit inbound connections from the internet and require all outbound traffic to pass through an explicit proxy. The pilot placed a single Broker VM at headquarters; all clinic syslog is forwarded over a 20 Mbps WAN link to that broker. After two weeks, 40% of endpoint telemetry is delayed by more than 30 minutes, the broker's CPU sits at 85% at peak, and clinic logs arrive late or out of order. Peak ingestion is 180,000 EPS, but daily average is 45,000 EPS. The team wants high availability and a design that can survive a single broker failure without losing log collection. They also need to know which firewall flows to request before the production rollout.

1. What are the most likely causes of the delay and reordering, and what architecture changes would you recommend? Justify each change.

Read the worked answer

Separate the two telemetry paths before changing hardware. Cortex XSIAM endpoint agents normally send telemetry to the tenant over TLS 443; they do not flow through the Broker VM unless they are explicitly proxied. So the endpoint delay points first to the HQ egress proxy, WAN saturation, or tenant-side ingestion backlog, while the clinic log delay points to the single broker and the 20 Mbps WAN. The broker at 85% CPU at peak is undersized for the aggregate parse/forward load and has no failover; store-and-forward over a congested WAN explains late and out-of-order clinic logs.

Recommended changes:

  • Deploy a Broker VM at each clinic so local sources are collected and parsed locally; only normalized events traverse the WAN/tenant path. This removes the HQ aggregation bottleneck and reduces WAN sensitivity.
  • Size each broker for local peak EPS with 30-50% headroom, not for the 180k aggregate. Keep the HQ broker for HQ sources only.
  • Convert each site to an HA pair with a VIP so a single broker failure does not stop collection; verify the HA design with the vendor's supported clustering method.
  • Add proxy capacity and allow direct egress from brokers to the tenant; do not route endpoint telemetry through an undersized HQ proxy if split tunneling is permitted.
  • Fix time sync with NTP at sources and brokers; timestamp skew worsens reordering.

Validate with a peak-load test and a broker failover test, watching dropped events, queue depth, and CPU below roughly 70%.

2. List the network flows and identity requirements. Include direction, purpose, and any risky exceptions.

Read the worked answer

Required flows, all initiated from the internal side:

  • Log sources to local Broker VM: TCP 6514 for TLS syslog, or TCP/UDP 514 only if the source cannot do TLS; internal only. Purpose: event collection.
  • Broker VM to the Cortex XSIAM tenant through the explicit proxy: TCP 443 outbound. Purpose: registration, configuration, and normalized event upload.
  • Endpoint agents to the Cortex XSIAM tenant through the proxy: TCP 443 outbound. Purpose: endpoint telemetry and response channels.
  • Broker HA peers: internal heartbeat/replication on the vendor-supported ports. Purpose: failover.
  • NTP and DNS from brokers and sources to internal services. Purpose: correct _time and name resolution.

Identity requirements: a dedicated broker registration/service account per site, a separate automation service account, API keys stored in the credential store, and least-privilege roles for administrators. Use MFA for human admins.

Risky exceptions to avoid: any inbound internet flow to a broker; broad 443 to any destination rather than an allowlist of tenant FQDNs; and TLS break-and-inspect on tenant or broker traffic if certificate pinning is used. If inspection must be bypassed, document the exception, scope it to the tenant FQDNs, and require security approval.

3. How would you size and validate HA? Include headroom, failure testing, and data-loss checks.

Read the worked answer

Size by site and by source, not by a single global EPS number. Inventory peak EPS per site, including syslog burst behavior and parsing complexity; JSON parsing and multiline merging cost more CPU than simple delimiter parsing. Apply 30-50% headroom above measured local peak, then confirm with the vendor sizing guidance. For HA, use an active/standby or supported cluster pair per site with a VIP, and ensure sources can reconnect to the VIP rather than a fixed broker IP.

Validation plan:

  • Run a representative peak load, not just a synthetic average, and record CPU, memory, disk queue, and dropped-event counters.
  • Fail the active broker during peak and confirm sources reconnect to the standby within the agreed RTO.
  • Compare source-side event counts with dataset counts for the same window to estimate loss; use a known high-volume source as a canary.
  • Confirm parsing and enrichment continue after failover, and that no duplicate storm is created when the failed node returns.
  • Test failback and document the procedure. Monitor queue depth after failover, because a standby that immediately receives the full backlog can saturate.