A 2,700-Indicator Enrichment Playbook Taking 90 Minutes
A daily enrichment playbook processes a threat feed: 1,500 IPs, 800 domains, and 400 file hashes. It uses a single script task that loops over each indicator and calls demisto.executeCommand for the corresponding reputation command, one indicator at a time. Total runtime is 70 to 90 minutes. The war room shows occasional "Timeout waiting for response" errors, and the investigation context reaches roughly 85 MB. The threat intel integration permits 300 requests per minute, returns about 8 KB of JSON per lookup, and bills per call. The playbook runs on a single engine configured with four workers. Analysts only need a verdict and a score in the layout, not the full raw response. Some indicators appear in more than one feed section.
1. Refactor the enrichment so it completes in a reasonable time without exceeding the API limit. Show the reasoning behind your target runtime.
Read the worked answer
Do the arithmetic first. 2,700 indicators at 300 requests per minute is a theoretical floor of about 9 minutes if every request were saturated and perfectly parallel. Running 70 to 90 minutes means the serial loop is spending most of its time on script and platform overhead per call, not on the API.
Three changes, in order of impact. First, deduplicate the input before any call. Indicators appearing in multiple feed sections are common, and de-duplicating is free - it reduces both runtime and billed calls, which matters because the integration charges per call. Expect a meaningful reduction from this alone.
Second, batch or parallelize. If the integration exposes a multi-indicator or bulk command, use it; that is always better than fanning out. If not, run the script with a bounded thread pool - around 20 workers is plenty - with a shared token bucket capping aggregate throughput at roughly 250 to 280 per minute to leave headroom for other playbooks using the same token. Do not simply raise playbook-level parallelism and hope: with four engine workers, parallel playbook tasks compete for the same four slots, so threads inside a script are the more controllable lever.
Third, cache within the run. Keep a dict of indicator to result so a repeat never triggers a second billed call.
Success criterion: 9 to 15 minutes for the full feed, no 429s, and billed calls roughly equal to unique indicators.
2. Context is at 85 MB and the engine throws timeout errors. What is causing each symptom and how do you fix them?
Read the worked answer
Both symptoms trace to the same design: every executeCommand result is written into context in full. 2,700 calls at roughly 8 KB each is over 20 MB of raw JSON before platform overhead, which explains the 85 MB figure, and large context payloads slow every downstream read and write.
Fix the context first. Set raw-response to false and pass a using argument that maps only the two fields you need, or better, extract verdict and score inside the script and write them to context with setContext under a compact key, returning no raw results at all. Add a DeleteContext step for the raw keys after extraction, scoped to those keys - never all, which drops state later tasks depend on. Target context under a few megabytes.
The timeouts are a consequence, not an independent fault. Large results increase the round-trip and serialization time per command, and at some point the platform's command timeout is hit. Reducing payload size will remove most of them. For the remainder, distinguish a genuine upstream timeout from a platform timeout: if the API responded slowly, retry with backoff; if the platform timed out while carrying a large result, the payload fix is the answer.
Finally, request only what the layout shows. Pulling full raw responses that no analyst ever views is pure cost - in latency, in context size, and in billing.
3. How do you handle failures and rate limiting so one bad indicator does not abort the run or blow the per-call budget?
Read the worked answer
Treat failures by class. A lookup that returns "unknown" is a valid result, not an error - record it as an unknown verdict rather than retrying, otherwise you pay repeatedly for the same non-answer. A transient error (429, 5xx, connection reset) deserves a bounded retry with exponential backoff and jitter, respecting Retry-After. A permanent error (401, 403, malformed request) should fail that indicator, be logged with the indicator value, and continue the batch - one bad hash must not abort 2,699 good lookups.
For rate limiting, keep the token bucket in the script and also set the integration instance's request cap just under the documented 300 per minute. Two independent limits mean a burst from a concurrent playbook cannot push the shared token over the edge. When the bucket is empty, block briefly rather than firing and absorbing 429s; if the wait would be long, checkpoint progress and exit the task cleanly so the run can resume.
Budget control: because the integration bills per call, deduplicate first, then skip indicators that already carry a recent verdict from a prior run - no need to re-bill the same IP every day if the score is unchanged and recent. Track calls made, calls saved by cache, and failures as context outputs so the cost trend is visible.
Finally, make the run resumable. Persist the list of completed indicators so a mid-run failure restarts from the remainder rather than re-billing everything.