XSOAR at 35 Tenants: Rate Limits, Campaign Storms, and Nondeterministic Verdicts

Diagnosis: the playbook is running per report, but the unit of work is the email and, above that, the campaign. Three hundred reports of one message is one investigation plus 299 pieces of corroborating evidence. Every duplicate incident costs six minutes of playbook execution, a ticketing call budget, and an SLA clock that is measuring noise.

Redesign

1. Deduplicate at ingestion. Compute a stable key from the message (message-ID plus a hash of the body and attachment set) and suppress duplicate incidents before the playbook runs, rather than inside it. Reports from additional users attach to the existing incident as additional reporters - that count is itself a useful severity signal.
2. Two-level incident model. One parent incident per campaign or per unique message, with child incidents or a linked-report list per reporting user. The heavy enrichment and detonation tasks run once on the parent; the children carry attribution and reporter follow-up only.
3. Pre-processing rules for routing. Bulk-reported, already-known-bad messages go straight to a verification queue instead of the full 38-task path. Distinguish "new message, needs detonation" from "known campaign, needs confirmation."
4. Split the playbook. Enrichment and verdict as one sub-playbook, notification and closure as another. That lets you run only the closure path on duplicates.

The commercial objection: billing per incident is a pricing construct, not an architecture requirement. Reframe with numbers - 300 incidents at six minutes each is 30 hours of engine time and 3,300 ticketing calls for one decision, which is both a cost and an SLA risk. Propose charging per unique investigation, or per reporter, and show that the tenant's bill reflects work actually performed. If the account team still resists, take it to the pricing owner as a commercial change, not a technical one - do not quietly keep the volume.

Why the current pattern is harmful: retrying a 429 three times with no delay is not a retry strategy, it is an amplifier. Each throttled call becomes four calls in a tight window, so the tenant's request rate goes up precisely when the API is asking it to go down. The retries also burn task time against a 15-minute SLA and can produce partial state - incident created, ticket not created, no record of the divergence.

Rework

1. Classify errors before acting. 429 and 5xx-with-retry-after are transient; 400/401/403 are terminal and must fail fast with a clear war-room note. Blind retry on a 403 just hides a credential problem.
2. Honor the server's signal. Where the response carries a retry-after value, sleep that long. Without one, use exponential backoff with jitter so 35 tenants do not synchronize into a thundering herd at the top of the hour.
3. Cap concurrency at the integration instance. Configure the instance's per-minute request budget below the tenant limit (for example ~240 of the 300 available) so normal bursts do not hit the ceiling at all. Leave headroom for manual analyst actions, which share the same quota.
4. Batch the calls. The 38 tasks likely make several single-record calls that can be one bulk request. Measure calls-per-incident before and after; that ratio is the real efficiency metric.
5. Make throttling observable. Emit a metric for 429 rate and queue wait per tenant. A rising 429 rate during Monday peak is a capacity signal the platform team should see before tenants complain.
6. Idempotency. Every create operation needs a dedup key so a retry after an ambiguous timeout does not create a second ticket. This is the failure mode that survives all the tuning.

Sequencing note: fix idempotency and error classification first - they are cheap and stop data corruption. Then tune concurrency and batching, which require load testing against a real tenant limit.

Why it matters: in a multi-tenant SOC, the verdict is the product. Two verdicts for one email means the outcome depends on which analyst's queue it landed in or which enrichment call happened to succeed - that is a reproducibility failure, and it destroys the tenant's ability to trust either answer. The overwritten classification is the same disease: the system records a conclusion that no one actually reached.

Nondeterministic verdict

• The root cause is almost always that enrichment runs per incident and depends on external state - a reputation lookup that changed, a sandbox that timed out, a vendor that returned partial data. Cache the enrichment result keyed to the message hash, with an explicit time-to-live, so a second report of the same message reuses the first decision rather than re-deriving it.
• Record the evidence set alongside the verdict: which lookups ran, what they returned, which failed. A verdict with a failed detonation step must be marked as degraded, not silently closed as benign.
• Where a second report contradicts the first, route to a re-review path rather than letting the newer result overwrite the older. The disagreement is a signal.

Field overwrite

• Identify whether the close task is writing a literal default value or re-setting a field from a task input. Playbook tasks should never hardcode values into fields that earlier tasks own.
• Set the classification once, from a single task, and have downstream tasks reference it rather than assign it. Where a field must be written late, use a mapping that only fills empty values rather than unconditionally overwriting.
• Add a validation step before closure: if classification is empty or contradicts the verdict, block closure. That one guardrail catches a whole family of playbook bugs across all 35 tenants.