Nightly Queue Collapse After Adding Ten Digital Workers
A shared services centre runs a claims-registration process on 12 Blue Prism digital workers, each with 4 GB RAM and 2 vCPUs, connected to a single runtime resource host. Historically 8 workers processed 6,000 queue items per night in 5 hours. After adding 4 more workers, throughput fell to 4,200 items and the nightly window now overruns by 90 minutes. Database CPU sits at 45%, but the session log shows repeated 'could not obtain a lock' retries averaging 12 seconds per item, and queue item load time has tripled. The team suspects the new workers are underpowered and proposes upgrading each to 8 GB RAM and 4 vCPUs.
1. Diagnose the most likely cause of the throughput drop and evaluate the proposed hardware upgrade.
Read the worked answer
The symptoms point to contention on the queue's work-queue locking rather than compute starvation. Database CPU at 45% and low per-worker utilisation argue against a hardware bottleneck. The decisive clue is the 12-second average spent retrying lock acquisition: with 12 workers pulling from the same queue, the probability of two sessions selecting the same item rises, and each collision costs a full retry cycle. That overhead consumes the very time that should be spent processing, so adding workers past the contention point reduces net throughput.
Hardware upgrades address CPU and memory pressure, which the evidence does not show. Spending on 8 GB/4 vCPU per worker would likely change nothing and could mask the real issue. The architect should instead:
- Confirm the contention hypothesis by measuring lock wait time and retry counts per session across a controlled run.
- Reduce the queue's contention by partitioning work: separate queues by claim type, region, or batch, so each worker pool draws from a smaller pool of items.
- Ensure item selection uses short transactions and releases locks quickly; avoid holding a lock while waiting on external systems.
- Re-test throughput at 10, 12, and 14 workers to find the real saturation point.
Only after contention is removed should hardware be revisited, and then only if per-worker CPU or memory metrics justify it.
2. Recommend a capacity-planning approach so the next scale-out decision is evidence-based.
Read the worked answer
The team needs a throughput curve, not a single data point. A defensible approach:
- Define a baseline unit of work: average processing time per queue item, broken down into application interaction, queue lock wait, and idle time.
- Run a controlled ramp: measure items per hour at 4, 8, 10, 12, and 14 workers under identical data volumes, holding the environment constant.
- Plot the curve and identify the knee where marginal throughput per added worker approaches zero. That knee is the practical capacity limit for the current architecture.
- Record the binding constraint at each point (queue locking, target application response time, database contention, licence count).
- Set a headroom target, for example run at 70-80% of the knee, so peak nights do not tip into contention.
Capacity planning should also account for business volume growth and the nightly window. If the knee is at 10 workers and demand will grow 20% annually, the architect must plan a structural change (queue partitioning, a second process split, or application-side batching) rather than simply adding workers. Document the curve and revisit it after any change to the process, the target application, or the infrastructure, because the knee moves.