The Uneven Worker Pool
A company runs Alteryx Server with a Controller node and five Worker nodes. Three workers are 16-core machines, two are 8-core machines. The default worker assignment is used, so workflows are distributed to any available worker. A set of heavy spatial workflows that use about 30 GB of memory and run for 45 minutes are intermittently failing with out-of-memory errors, but only sometimes. When they fail, they often fail on the two 8-core workers. The 16-core workers have 64 GB RAM; the 8-core workers have 32 GB RAM. The queue also contains hundreds of short workflows that complete in under a minute. The admin wants to stop the OOM failures without starving the queue of short jobs.
1. What is the most likely root cause, and how should the worker assignment be changed?
Read the worked answer
The Controller assigns work based on the default worker assignment, which sends jobs to any available worker. Because the 8-core workers have only 32 GB RAM, a 30 GB spatial workflow plus the engine overhead will exceed available memory, causing OOM errors. The intermittent nature is explained by the fact that the job only fails when it lands on a small worker; if a large worker picks it up first, it succeeds.
The fix is to create a dedicated worker assignment (or worker tag) for the heavy spatial workflows and assign only the three 16-core/64 GB workers to it. The short workflows can continue to use the default assignment, which includes all workers, or be assigned to the two small workers to keep the large machines free for heavy jobs. This isolates resource-heavy work from the general queue without reducing overall throughput for short jobs.
2. A colleague suggests simply increasing the number of workers to reduce queue time. Why does that not solve the OOM problem?
Read the worked answer
Adding workers increases parallel capacity, but it does not change which worker a given job can land on. If the new workers are also 8-core/32 GB machines and are included in the default assignment, the heavy spatial workflows will still sometimes be routed to an undersized machine and fail. The problem is a placement and resource-fit problem, not a throughput problem. Adding workers may even increase the failure rate by creating more chances for a heavy job to land on a small node. The correct approach is to constrain heavy jobs to workers that meet their memory requirements, then add capacity to that pool if queue time is still an issue.