CRM synchronisation: pagination, rate limits and unknown region codes
MediTrack syncs customers from a SaaS CRM. Every 15 minutes a scheduled event calls GET https://crm.example.com/v2/customers?updatedSince={lastRun} with a bearer token stored in a constant. The response contains 500 records per page and a nextCursor field that the current implementation ignores, so only the first page is ever imported. The API returns HTTP 429 roughly once an hour; the call activity's error handling is set to Rollback, which aborts the whole sync and skips the step that updates lastRun. The import mapping creates new Customer objects unconditionally, and last month's logs show 1,900 duplicate customers. Some CRM records reference a Region code that does not exist in Mendix, and those records throw a validation error that kills the batch. A recent CRM release renamed the JSON field customer_name to name; the import mapping no longer matches, and the sync reports zero imported with no error.
1. Design the retrieval and resilience part of the sync: pagination, 429 handling and the lastRun watermark.
Read the worked answer
Pagination. Treat nextCursor as mandatory. Loop: call the endpoint with the cursor, process the page, read nextCursor from the response, stop when it is empty. Each page is its own iteration so a failure costs one page, not the whole run. Never rely on offset paging against a system that changes while you read it; a cursor is stable, an offset is not.
429 and transient failures. Set custom error handling on the call activity so a 429 does not roll back the transaction and kill the run. Read the Retry-After header when present, otherwise back off exponentially (for example 1, 2, 4, 8 seconds) with a small cap, and retry a bounded number of times, for example five. Persist the retry state so the scheduled event does not start a second overlapping run: a SyncRun entity with status Running and an if-guard at the start of the microflow gives you a lock. Reduce the page size if 429s cluster, since rate limits are usually per minute.
Watermark. Update lastRun only after a page has been processed and committed, and store it with a deliberate overlap of a few minutes. Overlap means the same record may arrive twice, which is harmless once the upsert is idempotent; a watermark that is too tight silently loses records written during the run. Also handle clock skew: the CRM's updatedSince is presumably UTC, so never use the server's local time.
State the trade-off: retries and overlap increase duplicate traffic. That is acceptable only because the write path is made idempotent in the next step.
2. How do you make the import idempotent so the 1,900 duplicates cannot recur?
Read the worked answer
Give every customer a stable external key and upsert on it.
- Store the CRM identifier, for example CrmCustomerId, on Customer as a unique-ish attribute (Mendix does not enforce uniqueness in the model, so enforce it in the microflow and validate on import).
- Replace the unconditional create with: retrieve Customer where [CrmCustomerId = $importedId]. If it exists, update the fields; if not, create it. Alternatively use the import mapping's find-by option: mark CrmCustomerId in the mapping as the key, and the mapping resolves existing objects instead of creating new ones, which also works for associations such as Region.
- Do the whole thing in one transaction per customer so a failure halfway cannot leave a half-updated record.
- Keep a natural-key fallback for records that predate the identifier, and run a one-off deduplication microflow on the existing 1,900 duplicates before the next run: match on email or name plus region, merge into the oldest record, and log every merge for support.
Why the mapping matters: an import mapping without a find-by key is a create-only pipeline, so replaying any window duplicates everything. Idempotency is a property of the write path, not of the scheduler; no amount of retry tuning fixes it. Test it by running the same payload twice and asserting the customer count is unchanged.
3. Region codes that do not exist in Mendix kill the batch, and a renamed JSON field silently produced zero imports. What does that tell you about the sync's design?
Read the worked answer
Both symptoms are the same defect: the sync has no per-record error isolation and no observability.
Error isolation. A missing Region is a data-quality problem, not a system failure, and it must not abort a batch of 500. Split the per-record work into a sub-microflow with custom error handling so a failure rolls back only that record, and write a row into a SyncError entity: timestamp, external id, error type, raw payload, message. Then either leave the record unprocessed for a human or import it with a placeholder region and flag it. Never use a single validate-then-continue pattern that lets one bad record stop the loop.
Observability. A renamed field produced zero imports with no error because the import mapping found nothing to match and the microflow did not check the outcome. Add: a count of received versus imported versus failed records per page; an alert when imported is zero while received is positive; and a schema check that logs when an expected field is absent in the response. Keep the raw response body for a sample page when the count mismatches.
Process. Agree with the CRM owner on versioned endpoints and a deprecation window, pin the API version in the URL rather than relying on the latest, and add a contract test that runs against a recorded sample response so a rename fails in acceptance, not in production at 03:00.