Nightly invoice import: browser timeout, rollbacks and duplicate registrations
OrderFlow Support built an invoice-processing app. A button on an admin page starts microflow ACT_ImportInvoices. It retrieves 5,000 Invoice objects with status New, then loops: for each invoice it calls a REST endpoint that registers the invoice in the finance system, sets status to Registered and commits with events enabled. The same microflow is also triggered by a scheduled event each night. Four symptoms: (1) from the admin page the browser shows a generic error after roughly five minutes and the user gets no feedback; (2) when invoice 3,412 returns HTTP 400, the run stops, finance has already registered 3,411 invoices, and the next run registers them again, so duplicates accumulate; (3) the nightly run occasionally ends in an out-of-memory error; (4) audit finds invoices that are Registered in finance but still New in Mendix.
1. Diagnose the browser error and the memory problem, and describe how you would restructure the processing.
Read the worked answer
The browser error is a client timeout, not a bug in the loop. A microflow triggered from a page runs synchronously in the user's request. When it exceeds the client timeout the browser gives up and shows a generic error, while the server may keep working. A five-minute, 5,000-iteration job with a network call per iteration was never a page action.
Restructure:
- Trigger from a scheduled event or an asynchronous path, never from a page button. For interactive use, let the button create a Batch entity with status Queued and commit it, then show progress by polling that record. The user gets feedback and no timeout.
- Batch the retrieve. Retrieve in chunks of 200 to 500 with an offset loop (or retrieve by status with a limit), instead of loading all 5,000 invoices plus their associations into memory at once. The out-of-memory error comes from the whole set being held in the microflow's memory for the entire run.
- Avoid the per-invoice round trip if the API allows it. A bulk endpoint that accepts 100 invoices per call cuts the network cost by two orders of magnitude and shrinks the run to seconds.
- Commit outside the hot loop or with events disabled. Committing with events enabled fires all subscribed event handlers (and the object is re-validated) on every one of 5,000 iterations. Commit without events inside the batch, and commit the batch result explicitly.
State the trade-off: an asynchronous design needs a status field, a lock so two runs cannot overlap, and a way to surface failures, but it is the only design that survives a 5,000-record workload.
2. A single failing invoice aborts the whole run and the retry duplicates data in finance. Design the error handling.
Read the worked answer
Two separate problems: blast radius and idempotency.
Blast radius. Move the per-invoice work into a sub-microflow, for example SUB_RegisterInvoice, and set custom error handling on the call activity inside the loop so a failure rolls back only that invoice's changes and the loop continues with the next one. Record the failure on the invoice itself (status Failed, an error message attribute, an attempt counter) or in an ImportLog entity, and summarise the counts at the end of the run. Without custom error handling, any error rolls back the entire transaction, which is exactly what symptom (2) shows: finance received 3,411 registrations while Mendix kept every invoice at New, because the commit inside the loop was part of the same transaction and got rolled back with it.
Idempotency. Never call an external system from a state that has no memory. Before the REST call, commit the invoice with status Sending and a generated correlation key, then call the endpoint with that key so finance can deduplicate on its side. After a successful response set Registered; after a timeout set Unknown and let a reconciliation step query finance by the key rather than blindly re-registering.
Validate before sending: check required fields in a validation microflow so the 400 at invoice 3,412 becomes a data-quality report instead of a runtime exception.
3. What causes the drift where finance shows invoices as registered but Mendix still shows them as New, and how do you reconcile it?
Read the worked answer
Cause: the external call and the local commit are not one atomic unit, and the local commit is not durable until the transaction ends. The REST call succeeds, finance records the invoice, then the microflow sets Registered and commits. If a later iteration throws, or the transaction is rolled back for any reason, the local commit inside the same transaction is undone while the external side effect cannot be undone. HTTP timeouts produce the same asymmetry: finance may have processed the request even though Mendix never received the response.
Reconciliation design:
- Persist intent before the call: status Sending plus a correlation key and a timestamp, committed as its own transaction so a later rollback cannot erase it.
- On timeout or error, set status Unknown instead of Failed, because Failed invites a blind retry that creates duplicates.
- Run a scheduled reconciliation microflow that queries the finance API for everything in Sending or Unknown state by correlation key and sets the local status from the answer. This runs on a separate schedule from the import so it does not widen the failure window.
- Count and log mismatches daily. A rising Unknown count is the early warning that the integration is timing out, long before audit notices.
Accept that exactly-once delivery across two systems is impossible without cooperation from the other side. The practical target is at-least-once delivery plus a deduplicating key plus reconciliation, and the correlation key is what makes the retry safe.