The Loop That Runs Fine in Testing
A Python job iterates over 180,000 order IDs, and for each one calls a REST endpoint to fetch line items, then writes results to a list that is inserted at the end. In the test environment with 500 orders it completes in 40 seconds. In production it has never finished; it is killed at the 2-hour limit with roughly 12,000 orders processed. The endpoint allows 20 requests per second per client. The job runs single-threaded. Memory climbs steadily until the kill. A colleague suggests "just add more workers" and points out the API limit is 20 rps, so 20 workers should be safe.
1. Estimate the throughput ceiling and explain whether 20 workers will hit it, using the numbers given.
Read the worked answer
At 20 requests per second, 180,000 orders need at least 180,000 / 20 = 9,000 seconds, or 2.5 hours, just in request time, before any latency or overhead. Observed production rate is about 12,000 orders in 7,200 seconds, roughly 1.7 rps, so the single-threaded job is far below the API ceiling. The bottleneck is per-request latency plus serialization, not the rate limit.
Twenty workers would in principle allow up to 20 rps, which still means 2.5 hours minimum. So "more workers" does not make the job finish faster than the rate limit allows; it only helps if the current limit is latency-bound, which it is. But 20 workers all firing at once will also burst above 20 rps and get throttled, so the colleague's reasoning ("the limit is 20, so 20 workers is safe") is wrong: safety requires a shared rate limiter, not a worker count that happens to equal the limit.
2. Identify the memory problem and propose a design that respects the limit and bounds memory.
Read the worked answer
The memory climb is the accumulating results list: 180,000 orders' worth of line items held in memory until the end, then inserted at once. At 12,000 orders the process is already near its limit, so the list is the leak, not a Python quirk.
The design I would use: a bounded queue of order IDs feeding a fixed pool of workers (say 8), each worker fetching one order and writing its rows to a staging file or batched insert of a few thousand rows, then discarding. A token-bucket limiter shared across workers caps aggregate calls at, say, 15 rps to leave headroom under the 20 rps ceiling. The job checkpoints the last completed order ID so a restart resumes rather than re-fetching.
With 8 workers and 15 rps, the floor is still 180,000 / 15 = 3.3 hours, so I would raise the concern that the 2-hour limit may be infeasible for this workload and ask whether the limit can move or the order set can be filtered to changed orders only. That question matters more than the worker count.