Enrichment API Rate Limit Causes Fail-Open Containment
A Cortex XSIAM playbook enriches every new issue by calling three external services: a sandbox, a threat-intelligence API, and an internal asset database. The threat-intelligence vendor allows 1,000 requests per hour per tenant. The tenant creates 2,800 issues per day, and the playbook currently makes four threat-intel lookups per issue (source IP, destination IP, file hash, and domain). About 35% of indicators repeat within the same hour. During a recent surge of 900 issues in 40 minutes, the playbook hit HTTP 429 responses. It has no retry logic; failed enrichment tasks are marked 'complete' with empty outputs, and the playbook continues to a containment decision that treats empty reputation as 'unknown' and does not contain. Analysts found 14 malicious issues that were not contained for up to 3 hours. The SOC wants containment decisions to be safe under load and to avoid API overages.
1. Why did containment fail, and how should the playbook handle missing enrichment?
Read the worked answer
The playbook failed open. When the threat-intel call returned HTTP 429, the enrichment task had no retry logic and was marked complete with empty outputs. The downstream decision treated empty reputation as 'unknown' and chose not to contain. That is a logic error: absence of enrichment is not evidence of benignity. Fix by defining explicit states: enriched-malicious, enriched-benign, and enrichment-unavailable. For high-severity issue types such as ransomware or credential theft, enrichment-unavailable should trigger containment or a manual urgent task, not pass through. For low-severity issues, it can queue for review. Disable 'continue on error' or branch on task error, and add a field such as enrichment_status that the decision task must check. Safe defaults must be risk-based, because auto-isolating every unknown would create its own outage.
2. Redesign the enrichment flow to stay within the API limit during a surge.
Read the worked answer
Redesign around dedupe, cache, and rate control. Build a unique indicator list across the issue and recent issues, cache reputation for 24 hours, and batch lookups so four calls per issue become one call per unique indicator. At 2,800 issues/day, raw calls are 11,200/day, which is under the daily limit but not under the hourly limit during a surge. Add a token-bucket limiter at about 900 requests/hour to leave headroom, queue overflow, and retry 429 with exponential backoff and jitter. Prioritize high-severity issues and never-expired indicators. Use local asset data for internal IPs instead of external API calls. Record cache hits and queue depth so the design can be tuned. This keeps the vendor under 1,000/hour even when 900 issues arrive in 40 minutes.
3. How would you test the redesigned playbook under realistic failure conditions?
Read the worked answer
Test with a replay of the 900-issues-in-40-minutes surge against a lab tenant or a mocked integration. Inject 429 responses and empty outputs, then verify the playbook does not mark enrichment complete or skip containment on high-severity issues. Measure containment latency, 429 rate, queue depth, cache hit rate, and false containment. Add a chaos test where the threat-intel API is fully down for 30 minutes. Monitor production with alerts on enrichment failure rate, backlog age, and any high-severity issue that reached a containment decision with enrichment_status=unavailable. The acceptance criterion is that a surge degrades enrichment speed, not containment safety.