Nightly Invoice Run Misses Its Window After a Queue Spike
A finance shared-services team runs a Blue Prism process that posts supplier invoices into an ERP system. The work queue receives between 2,000 and 2,600 items each night, and the operations SLA requires all items to be attempted before 06:00. The process has 4 runtime resources, each configured with a maximum of 2 concurrent sessions. Average handling time per item is 35 seconds, but 8 percent of items hit a transient ERP lock and currently throw a System Exception that the process does not retry; those items land in the Exception queue and a human clears them the next morning, which now takes 3 hours daily. Last week the queue was not fully attempted on three nights, and the backlog rolled into the next day. The team wants to raise concurrency to 4 sessions per resource and add automatic retries.
1. Assess whether raising concurrency to 4 sessions per resource will close the capacity gap, and identify the more reliable levers. Show your arithmetic.
Read the worked answer
First establish the raw capacity. With 4 resources at 2 sessions, there are 8 concurrent sessions. At 35 seconds per item, each session handles about 103 items per hour (3600/35), so the fleet handles roughly 823 items per hour. A 2,600-item night needs about 3.2 hours of perfect throughput - well inside an 8-hour window on paper, so the shortfall is not raw concurrency but lost time: the 8 percent System Exceptions (about 208 items) are removed from the queue and reworked manually, and any session that dies mid-item loses its work.
Doubling to 16 sessions would nominally lift throughput to about 1,646 items per hour, but that only helps if the ERP lock is not contention-driven. Transient lock failures usually get worse as concurrency rises, so the change may convert an 8 percent failure rate into a higher one and simply move the bottleneck. It also multiplies licence and infrastructure load without addressing the 3 hours of daily manual clearing.
Better levers, in order: (1) classify the ERP lock as a Business Exception or System Exception correctly - a transient lock is a System Exception and should be retried in-process with a small backoff before the item is marked failed; (2) configure queue retry logic so failed items are automatically re-queued rather than sent to manual clearing; (3) check whether the 35-second average hides a slow step that can be batched or cached; (4) only then tune concurrency, and measure failure rate at each step. Retries plus correct exception classification recover most of the 208 daily items and remove the manual backlog, which is the real cause of the rolled-over queue.
2. The team proposes a blanket retry of 3 attempts on every stage in the process. What is wrong with that, and what retry design would you recommend?
Read the worked answer
A blanket retry on every stage is dangerous because it retries actions that are not idempotent and it hides genuine defects. If the invoice post actually succeeded in the ERP but the response timed out, a retry creates a duplicate posting. If a stage fails because of a data problem - a missing cost centre, a malformed supplier code - retrying three times just triples the failure count and delays the item reaching a human.
A sound design retries only at the boundary where transient failure is expected and where the operation can be made safe. For the ERP post, either use an idempotency key or check-before-write: query whether the invoice reference already exists before posting, and treat an existing match as success. Wrap that single action in a retry with a short, bounded backoff (for example 2 attempts at 10 and 30 seconds) and let the queue's own retry mechanism handle items that still fail, so the retry count and reason are recorded on the item. Leave deterministic validation stages unretried - they should fail fast and route to a Business Exception with a clear reason. Finally, instrument the retry: log which stage, which attempt, and which error text, so the team can see whether the lock rate falls once concurrency is tuned. Retry policy is a design decision per integration point, not a global switch.