The 3 a.m. Queue Collapse: When a Bot Pool Outgrows Its Runner
A shared unattended runner (8 vCPU, 32 GB RAM, one Control Room device) hosts 14 production bots. Each bot is configured with a maximum of 3 concurrent sessions, so peak concurrency reaches 22 sessions on 8 cores. For three weeks, jobs queued between 01:00 and 04:00 have intermittently failed with 'session terminated unexpectedly' and occasional out-of-memory errors. Re-running the same jobs manually at 09:00 succeeds every time. The Control Room shows all devices online, and per-bot success rates are 99%+ outside the batch window. The ops team's proposal is to raise the per-bot concurrency cap to 5 to clear the backlog faster.
1. Diagnose the most likely root cause of the overnight failures and evaluate the ops team's proposed fix.
Read the worked answer
Diagnosis: The symptoms point to host-level resource exhaustion, not bot logic. 22 concurrent sessions on 8 vCPU means roughly 3 sessions per core; each Automation Anywhere (AA) session carries its own runtime overhead (browser/UI automation processes, memory per session). Overnight batch overlap pushes memory and CPU past the host's capacity, so sessions are killed by the OS or by the runtime's own watchdog - hence 'session terminated unexpectedly' and OOM. The 09:00 success is the tell: same artifacts, different load. This is a capacity and scheduling problem, not a correctness problem.
Evaluate the fix: Raising the cap to 5 increases peak concurrency (up to 70 sessions) and makes the failure mode worse, not better. It treats a queue-depth symptom with a throughput lever the host cannot support.
Better direction: (1) Right-size - calculate sessions-per-host empirically (start at 1–2 per vCPU minus headroom), and cap total concurrent sessions at the device, not just per bot. (2) Stagger schedules so batch bots don't overlap. (3) Add a second runner/device and let the Control Room load-balance across them. (4) Only after headroom exists should anyone discuss raising concurrency.
2. The team insists the bots are 'simple' and should not need more hardware. How would you prove or disprove that claim before approving new infrastructure spend?
Read the worked answer
Measure, don't argue. Propose a short evidence-gathering plan:
- Per-session footprint: Run one instance of the heaviest bot in isolation and capture peak RSS and CPU for the session process tree. Multiply by realistic concurrency to estimate demand versus the 32 GB / 8 vCPU envelope.
- Overlap map: Pull the schedule and plot which bots' execution windows collide. If 22 sessions cluster into a 45-minute window, the average load is irrelevant - the peak is the problem.
- Correlate failures with load: Join failure timestamps against concurrency at that moment. If failures only occur above a threshold (e.g., >16 sessions), that is direct causal evidence.
- Cheap experiment first: Temporarily cap the device at, say, 12 total sessions and stagger two of the batch bots. If overnight failures stop, you have disproved the 'we need nothing' claim without buying hardware.
Present the threshold finding to the team. If a scheduling change alone holds, defer the spend; if failures persist below the safe threshold, the hardware case is justified with data.