Scalability, High Availability, and Performance
Scalability has two axes: vertical (more CPU/RAM on a node) and horizontal (more Worker nodes). Because the Controller is the coordination point, adding Workers increases throughput but does not remove the Controller as a bottleneck or single point of failure. High availability therefore requires Controller redundancy (failover configuration) and a MongoDB replica set so the database survives node loss. Workers are typically the easiest tier to scale and the least critical to make redundant, because a lost Worker only affects in-flight jobs.
Performance concepts: worker threads per node (too many causes memory pressure and context switching; too few causes queueing), memory for large workflows and spatial/sort operations, disk I/O for the persisted workflow directory and MongoDB, and network latency between Workers and the Controller/MongoDB. Worker tags let you segregate heavy or specialized workloads. Caching of apps and results, and result retention, affect both performance and storage.
Traps: assuming more Workers always means faster (the Controller and MongoDB can become the constraint); assuming multiple Workers equal HA (they do not-lose the Controller and the system is down); over-provisioning threads on a memory-constrained node; and ignoring that a slow MongoDB degrades everything.
Worked example: Throughput plateaus as you add Workers. Check Controller CPU and MongoDB latency. If MongoDB is saturated, adding Workers makes it worse. The fix is to scale the database tier (replica set, faster disks) before adding more Workers.
Rule of thumb: scale Workers for throughput, scale/failover the Controller and MongoDB for availability-and always find the current bottleneck before adding capacity.