Atlas Freight: a 60-second timeout and a partner who wants OData

Size the transaction. 4,000 shipments × 25 packages is 104,000 objects plus the full JSON string held in memory, all in one transaction and one commit. That is the timeout: not the HTTP call itself but a single enormous unit of work, with index maintenance and undo/redo logging around it.

Redesign:

• Use pages of 200 shipments (~5,200 objects). 4,000 shipments is then 20 requests - about 10 seconds of traffic at 120 requests/minute, comfortably inside the budget, and each commit is small enough to finish well under 60 seconds.
• Treat 429 as a normal control-flow signal: read Retry-After, wait that long, retry with exponential backoff and a capped attempt count. Microflows have no built-in sleep; implement the wait in a Java action, or better, persist NextAttemptAt and let a one-minute scheduled event pick the job up. A tight retry loop inside a microflow will make the 429s worse.
• Set an explicit HTTP timeout on the call so a hung partner does not hold a runtime thread indefinitely.
• Persist progress in a SyncState entity: partner cursor or offset, last successful page, timestamp. A failure resumes at page N+1. For throughput, process pages through a Task Queue with a small, configured concurrency - but only if the per-page idempotency guarantee below exists, otherwise parallel workers will race on the same records.
• Log per-page counts (received, created, updated, rejected) so the run can be reconciled with the partner's totals.

The principle: the unit of work should be the smallest thing you can safely commit and resume, not the largest thing you can hold in memory.

Diagnosis. The match key is the customer reference string, which is nullable and not guaranteed unique. Blank references match each other or match nothing depending on how the check is written, and a string that differs by whitespace or case will not match a stored value at the database level. Worse, the retry restarts from page 1, so every record in the already-committed pages is offered to the same weak check a second time. This is the classic combination: a non-unique business attribute used as identity, plus a retry that is not resumable.

Design:

1. Real identity. Use the partner's shipment identifier as the key, stored in a dedicated attribute with a Unique validation rule (a database-level unique constraint). If the partner has no identifier, derive a deterministic composite key and hash it; never leave the key blank - reject the record with an ImportError instead.
2. Upsert with a defined meaning. Retrieve by the key (indexed), update if found, create if not. The unique constraint is the backstop for races between parallel workers; catch the commit-level violation and treat it as 'already exists, update on next pass'.
3. Resumable retries. Combine with SyncState so a retry starts at the failed page. Idempotency and resume solve different halves of the problem, and you need both: idempotency makes re-processing harmless, resume makes it rare.
4. Reconciliation. A daily count comparison against the partner endpoint (or their report) catches the silent cases the constraint cannot, such as shipments the partner considers sent but which were never received.

State in the review notes that a unique constraint is the only guarantee that survives concurrency and restarts; microflow-level checks alone do not.

Do not publish the internal Shipment entity. Exposing it exposes every attribute, including margin data, and every association, including Invoice. OData clients can navigate associations with $expand, so an exposed association is an exposed data set - the partner would be one query away from invoices and margins. It also creates a permanent contract on an entity the internal team will keep changing.

Expose a projection. Build a dedicated read-only entity for the partner: either a persisted PartnerShipment kept in sync by a microflow, or - better, since it is read-only and always current - an OQL view entity that selects only the agreed fields (partner identifier, status, dates, destination, package counts) and, if the partner needs package lines, a second exposed entity for those. Give the partner a versioned service name so a future field change does not break them.

Security. Enable authentication on the published service (basic auth over TLS, or OAuth if the partner supports it; never anonymous), verify how the service applies entity access in your Mendix version and do not assume it - if access is not enforced per user, the projection itself is the control. Scope the exposed entity set to exactly the partner entity, and turn off exposure of anything else in the module.

Performance and disclosure risks to flag:

• With no default page size, the partner can request the entire set; 380,000 rows will be serialised and may exhaust the runtime. Enforce a maximum page size and document $top/$skip usage.
• $filter and $orderby on unindexed attributes turn every partner query into a table scan on your production database. Index the attributes you advertise as filterable or sortable, and test with the partner's real queries.
• $expand over one-to-many associations multiplies row counts; decide deliberately whether package lines are exposed at all, and if so, as a separate entity set rather than an expandable association.
• Agree on a change policy: additive fields are safe, renames and removals are breaking and need a new service version.