The 40-Second Window in a Batch Reconciliation
A nightly reconciliation process reads 6,000 invoice rows from a CSV, opens each matching record in a legacy ERP client, compares three amounts, and writes a variance flag. The ERP client shows a modal progress dialog after every save that clears in roughly 2.5 seconds but occasionally lingers 40 seconds when the database is under load. The current design uses a fixed 3-second Wait stage after each save. Last month the process failed on 11 nights out of 30, always with the same symptom: the next element is not found because the dialog is still on screen. Average nightly runtime is 4 hours 10 minutes, and the 6 a.m. reporting deadline leaves 50 minutes of slack. The client's IT team refuses to disable the dialog and will not grant database access.
1. Diagnose the root cause of the intermittent failures and explain why increasing the fixed Wait to 45 seconds is not an acceptable fix.
Read the worked answer
Root cause. The design treats a variable-duration event as if it had a constant duration. A fixed Wait of 3 seconds is a guess about how long the dialog will last, not a check that it has gone. On a loaded database the dialog outlives the guess, so the next navigation step executes against a screen that is still blocked. The failure is therefore a synchronisation defect, not a flaky application.
Why 45 seconds is wrong. Raising the Wait to cover the worst case makes every one of the 6,000 iterations pay the worst-case cost. That adds roughly 42 seconds × 6,000 ≈ 70 hours of dead time across the run - far beyond the 50 minutes of slack - so the process would never meet the 6 a.m. deadline. It also does not remove the defect: a 60-second stall would still break it, so you have traded a frequent failure for a guaranteed timeout.
The correct pattern. Replace the fixed Wait with an active wait: poll for the dialog's disappearance (or for the target element's existence) on a short interval, for example every 500 ms, with an overall timeout of 60 seconds. Exit the loop as soon as the condition is true. Normal iterations then cost about 2.5 seconds and only the rare slow ones cost up to 40 seconds. Expected runtime stays close to today's, worst-case stalls are absorbed, and a genuine hang raises a clean, named exception instead of a confusing 'element not found' three steps later. Add retry logic with a bounded attempt count and log the elapsed wait per iteration so the IT team can see the load pattern.
2. The business owner asks whether the whole process could simply be run twice, with the second run picking up failures. Evaluate that proposal.
Read the worked answer
What it would fix. A second pass would catch rows that failed on the first run, which sounds attractive because the failures are sporadic and affect a small fraction of rows.
Why it is the wrong answer here. First, the failures are not row-specific - they are caused by a timing race that any row can hit, so the second run has the same defect and will fail on a different random subset. You would be layering a probabilistic workaround on a deterministic bug. Second, the runtime budget forbids it: a full pass takes over four hours, and there are only 50 minutes of slack, so a second full pass cannot fit before the deadline. Third, a re-run needs a reliable way to know which rows already succeeded; without an explicit status column or work queue, the second pass would re-process completed rows and risk duplicate postings in the ERP - a data integrity issue, not just a performance one.
What to do instead. Fix the synchronisation as described, then add a targeted recovery path: mark each row's outcome in a work queue, and on failure re-queue only the failed items with a retry count. That gives the business the resilience they are asking for without a second full pass and without duplicate postings. If they still want a safety net, an exception queue reviewed each morning is cheaper and safer than a blind re-run.