Engine Saturation in a 40-Tenant MSSP Deployment
You administer a multi-tenant MSSP deployment: two XSOAR 8 application servers behind a load balancer, six engines split across two datacenters, and 40 tenant accounts. During a phishing wave, ingestion jumped to roughly 3,000 incidents per minute for 25 minutes. CPU hit 95 percent on four engines while the other two idled near 15 percent. Playbook tasks queued for 12 to 18 minutes and the SLA dashboard flagged 600 breaches. Investigation shows tenants are pinned to engines through a static tag list built two years ago; the three highest-volume tenants all point at the same two engines. Engine-04, which sits behind a NAT gateway, cannot reach two tenants' SIEM APIs at all, though those tenants' incidents still get created. The customer also demands that one tenant's playbooks never see another tenant's context. You have a 90-minute maintenance window and cannot buy hardware this quarter.
1. The load is wildly uneven (four engines at 95 percent, two at 15 percent). What is the most likely cause, and what is the first change you make inside the 90-minute window?
Read the worked answer
The load balancer is a red herring. Engines connect outbound to the application server and poll it for work; the LB distributes analyst UI/API traffic, not playbook execution. Execution placement is decided by tenant-to-engine assignment, and here it is a static tag list that pins the three loudest tenants to the same two engines. Round-robin is not happening at the execution layer at all.
So the fix is not "add engines" - adding two engines while every tenant stays pinned would leave the new ones idle. The first change is to introduce engine groups (for example tier-high-volume, tier-standard, tier-low) and re-map tenants into them so the top three tenants are spread across four engines instead of two. Then verify no integration instance is individually pinned to a specific engine, because instance-level pinning silently overrides group placement.
Second lever: the queue depth is also a function of how long each task holds a worker. Long-running enrichment tasks on an engine reduce effective capacity. During the window, check which playbooks dominate task time and consider moving the heaviest sub-playbooks to a dedicated group, or lowering per-engine worker count only where memory pressure exists.
Success check: after reassignment, per-engine CPU should converge toward the mean within one ingestion cycle. If it does not, the pinning is still in effect somewhere.
2. Engine-04 sits behind a NAT gateway and cannot reach two tenants' SIEM APIs, yet those tenants' incidents are still created. Explain why, and list the verification steps before you change firewall rules.
Read the worked answer
Incidents are created because fetch and classification happen on the application server side, not on the engine. The engine is only needed when a playbook task has to execute an integration command that talks to the SIEM. That is why ingestion looks healthy while enrichment fails - the failure surfaces as task errors in the war room, not as missing incidents.
The direction of connectivity matters. Engines initiate outbound connections to the application server; the application server does not dial into engines. So the NAT gateway must permit engine egress to the app servers on 443 and to the SIEM endpoints, and the SIEM's allowlist must contain the NAT public IP, not the engine's private address. If the tenant's integration relies on inbound webhooks or a listener, that design cannot work from behind this NAT at all.
Verification order: (1) confirm which engine actually ran the task, using the war room entry and the engine name, because an "any engine" instance may have executed it on a different engine and masked the problem; (2) run the connectivity test from inside the engine container, not from the host, since DNS and proxy environment variables differ; (3) check for a proxy that terminates TLS and breaks certificate validation; (4) test DNS resolution of the SIEM hostname from the engine; (5) only then request firewall changes. Pinning the integration instance to engine-04 first makes the test deterministic.
3. How do you enforce the requirement that one tenant's playbooks cannot see another tenant's context, and what would you audit first?
Read the worked answer
Tenant context isolation is not automatic just because tenants are separate records. The main leak paths are shared integration instances, shared credentials, and shared indicator/list data.
Design: give each tenant its own integration instances with its own credentials - never a single instance serving multiple tenants, because the instance context and cached tokens are shared. Assign tenants to engine groups such that execution does not co-mingle long-lived integration contexts. Keep indicator reputation scoped so a tenant's indicators are not enriched from another tenant's data; where the platform supports tenant-scoped indicators, enable it, and avoid global lists that contain tenant-identifying values. Enforce RBAC so tenant users and tenant service accounts cannot read other tenants' incidents, playbooks, or lists, and review the Administrator role holders, since that role crosses tenants by design.
Audit first: enumerate every integration instance and check whether it is shared across tenants; then enumerate global lists and indicator types and look for tenant names, hostnames, or API keys belonging to one tenant visible from another. Next, review playbooks that call SearchIncidents or GetIncidentsByQuery without a tenant filter - those are the most common accidental cross-tenant reads.
Finally, test the isolation rather than assume it: run a playbook as the lowest-privilege tenant account and attempt to read a known incident ID from a different tenant. A 404 or permission error is the expected result; anything else is an open finding.