Queue Items That Vanish on Exception
A claims-handling solution uses a Blue Prism work queue with 4,200 items loaded each morning. The design sets each item to 'Exception' when the downstream portal rejects a claim, and the operations team then reviews the exception queue manually. Recently the team reports that items are 'disappearing': the daily count of loaded items (4,200) does not match the sum of completed, exceptioned, and still-pending items, which totals about 4,150 on a typical day. Investigation shows the process sometimes throws an unhandled exception (a portal timeout) after it has already marked the item as complete but before it has written the claim reference back to the queue. There is also a second process that loads the queue and occasionally runs while the first is still processing. No item-level audit is currently kept outside the queue.
1. Account for the missing items and identify the design defects that allow them to go untracked.
Read the worked answer
Where the items go. The gap of roughly 50 items per day has two sources.
- Mark-complete-then-crash. The process marks the item complete, then performs a second operation (writing the claim reference) that can fail. When the timeout throws, the item is already in a terminal state, so it is neither pending nor exceptioned - it looks finished but its reference was never recorded. From the queue's perspective it is complete; from the business's perspective the work is unverifiable.
- Concurrent loaders. A second process loading the queue while the first is processing can add items mid-run, so the morning's 4,200 figure and the end-of-day totals are measured against different populations. Some of the apparent shortfall is a counting artefact, which is exactly why the discrepancy can't be pinned down without item-level audit.
Design defects. (a) The item is not treated as a unit of work with a single atomic completion point - status changes and data writes are split. (b) There is no external audit record, so the queue is both the work list and the only evidence, and any queue anomaly destroys the evidence. (c) There is no lock preventing two loaders from running concurrently. (d) The unhandled timeout is not converted into a controlled exception with a retry policy.
2. Recommend a redesign that makes every item traceable end to end, and state what you would add to detect this class of problem in future.
Read the worked answer
Make completion atomic. Reorder the work so all data writes (claim reference, outcome, timestamps) happen before the item is marked complete. Marking complete becomes the last, single action, and it should only run when the preceding writes have succeeded. If the portal times out, the item stays in progress and is caught by the exception path rather than being silently closed.
Convert timeouts into controlled exceptions. Wrap the portal call in a retry block with a bounded attempt count and a defined back-off. On exhaustion, mark the item Exception with a categorised reason code (timeout, rejection, data error) so the manual review team can triage by cause rather than by guesswork.
Add an external audit. Write an append-only log - one row per item per state transition, with item key, timestamp, process name, and outcome - to a database or file outside the queue. The queue remains the work list; the log becomes the source of truth for reconciliation. A nightly check should assert that loaded = completed + exceptioned + pending + in-progress and raise an alert on any variance.
Prevent concurrent loaders. Use a named lock or a single scheduled loader so only one process loads the queue at a time, and tag each item with its load batch so totals are always compared within a batch.
Detection for the future. The reconciliation check plus the audit log turns a silent 50-item leak into a same-night alert with a list of affected item keys.