Duplicate Incidents and Fetch Overruns in a Custom Scanner Integration
You maintain a custom integration that pulls vulnerability findings from an internal scanner for three business units. The API allows a page size of up to 200, paginates with offset, enforces 60 requests per minute per token, and returns findings sorted by lastSeen with one-second precision. Baseline volume is about 12,000 new findings per day, but the weekly scan produces roughly 25,000 findings in a single hour. fetch-incidents runs every minute. Analysts report two problems: duplicate incidents for the same finding, about 8 percent of volume, and intermittent "fetch did not complete" warnings during the weekly spike. The integration currently stores only a last_fetch epoch in integration context and fetches exactly one page per run. The scanner team refuses to add a unique monotonic ID to the API this year.
1. Design a fetch strategy that removes the 8 percent duplicates without a unique monotonic ID from the API.
Read the worked answer
The duplicates come from two compounding issues. First, last_fetch is stored as an epoch and compared against a one-second-precision lastSeen, so any run that lands inside a busy second either re-reads that second or skips it depending on whether the comparison is > or >=. Second, fetching a single page per minute at a fixed offset means the boundary moves between runs.
Do not rely on a scalar timestamp. Store a composite cursor in integration context: the last processed (lastSeen, finding_key) pair, and query with lastSeen >= cursor_time, then discard everything up to and including the stored key. That gives a stable, resumable boundary even when many findings share a second.
For deduplication, compute a deterministic composite key from immutable fields - host identifier, plugin/check ID, port, and first-seen date - hash it, and write it to a custom field on the incident. Then add a pre-processing rule or a dbotFindSimilarIncidents step keyed on that field so a repeated finding is linked or dropped rather than reopened as a new incident. Keep a bounded set of recently emitted keys (for example the last 10,000, evicted oldest-first) in integration context as a fast pre-check, but treat the incident-side key as the durable guard, because integration context is small and can be reset.
Finally, make the fetch idempotent: rerunning the same minute must produce zero new incidents. Test that explicitly by replaying a captured response twice.
2. The weekly scan produces about 25,000 findings in one hour. What has to change in the fetch loop, and where does offset pagination break?
Read the worked answer
25,000 findings in an hour is roughly 420 per minute, so with a maximum page size of 200 you need at least three full pages per minute just to keep pace - the current one-page-per-run design falls behind immediately and the backlog grows. That is the source of the warnings, not the API rate limit: 60 requests per minute is far more than the three per minute you need.
Change the loop to page until one of three conditions is hit: results exhausted, a wall-clock budget is reached (leave headroom under the 60-second interval, roughly 45 to 50 seconds of work), or the rate limiter signals throttling. Persist the next offset and the cursor in integration context so the next run resumes rather than restarting, and log pages fetched and findings emitted per run so backlog is visible.
The deeper problem is offset pagination itself. If new findings are inserted while you page, the result set shifts: rows move from page 2 to page 1 and you either re-read them (duplicates) or skip ahead (missed findings). With a lastSeen cursor plus a since parameter you avoid the shifting-window problem for anything older than the cursor, and the tie-breaker handles the boundary second. If the API only offers offset, sort deterministically and page by cursor time rather than by offset whenever possible.
Add backoff on 429 and 5xx responses with a capped retry, and treat a partial page as a signal to stop and resume next run rather than to advance the cursor.
3. How do you implement rate limiting and error handling so one slow hour does not stall the whole tenant?
Read the worked answer
Two layers, because the platform layer alone will not protect you. Set the integration instance's maximum requests per minute slightly below the hard limit - around 50 to 55 against a 60 limit - so the platform throttles before the scanner starts returning 429s, and other playbooks sharing that token do not get blocked by a burst. On top of that, implement a client-side token bucket inside the integration so the fetch loop paces itself and does not rely on hitting an error to slow down.
Error handling must distinguish classes. A 429 or 503 is transient: honor Retry-After when present, apply exponential backoff with jitter, cap total retries, and persist the cursor so a failed run does not lose position. A 401 or 403 is not transient: fail the command loudly, return an actionable error, and stop the loop, because retrying will just burn the rate limit. A malformed finding in an otherwise valid page should be logged and skipped, not abort the page - one bad record should not create a permanent backlog.
Avoid sleeping while holding an engine worker for long periods; if backoff exceeds a few seconds, end the run cleanly, save state, and let the next scheduled fetch continue. Finally, surface health: emit backlog count and last successful cursor time as context output so an analyst can see the fetch is behind before duplicates start appearing again.