Salesforce MuleSoft Developer: The Retry That Ate the SLA

The Retry That Ate the SLA

A synchronous Order Submission flow on an HTTP listener validates the payload, calls an Inventory API, calls a Customer API, then inserts a Salesforce record. Inventory returns 503 for roughly 5% of calls during its nightly maintenance window. To fix that, an engineer wrapped the entire flow body in an until-successful scope with maxRetries=3 and millisBetweenRetries=5000. The failure rate is now near zero, but p99 latency is 12 s and support reports orders inserted into Salesforce twice. The consumer contract says 2 s p99. Inventory exposes no idempotency key; Salesforce has an external ID field the team can use.

1. Diagnose the latency and redesign the retry.

Read the worked answer

Diagnosis: the retry boundary is far too wide. Each attempt re-runs validation, the Inventory call, the Customer call, and the Salesforce insert. On a bad path the request holds a thread for up to 3 x 5 s of sleeping plus the work itself - that is where 12 s comes from. The duplicate Salesforce records are the same bug seen from the data side: attempt 2 re-inserts a record attempt 1 already committed.

Redesign:

Wrap only the Inventory HTTP request in a try scope containing until-successful, with maxRetries=2 and millisBetweenRetries=250.
Set responseTimeout on the HTTP request (for example 1000 ms) so a single attempt cannot hang.
Compute the worst case and check it against the 2 s budget: 3 attempts x 1 s + 2 x 250 ms = 3.5 s - still over. Either drop to 1 retry with a 600 ms timeout, or call Inventory asynchronously and degrade gracefully when it is unavailable during the maintenance window.
Keep the Salesforce insert and Customer call outside any retried scope.

The principle: retry at the narrowest boundary that can actually recover, and always price the worst case.

2. How do you guarantee no duplicate Salesforce records when the client or the flow retries?

Read the worked answer

Use Upsert with the external ID field, configured with duplicate detection. An insert followed by a check-then-insert is racy under concurrency; upsert on a stable business key is not.
Derive the key from the request, not from a random id generated server-side on each attempt - otherwise a retry produces a new key and a new record.
Add an Object Store idempotency guard keyed on that business key with a TTL, so a repeated request short-circuits to the original result rather than re-entering the flow. Note the guard is a convenience, not the correctness proof - the upsert is.
Never place the write inside a retried scope. Retries must wrap only the call that is known to be idempotent or safely repeatable.

Also decide what the caller sees on a repeat: a 201 with the original identifier, or a 409. Document it in the spec.

3. Inventory succeeds but Customer returns 500. What are the semantics, and how do you implement the choice you make?

Read the worked answer

First decide the business rule: is the Customer lookup enrichment or a prerequisite?

If it is enrichment (the order can proceed without it):

Put the Customer call in a try scope with on-error-continue, logging the failure and setting a customerResolved=false variable. The flow completes and the response notes the degraded enrichment.
Do not use scatter-gather to make the calls concurrent without understanding its failure model: if any route errors, the whole scope errors. To get per-route tolerance you must put on-error-continue inside each route.

If it is a prerequisite (an order without a validated customer is invalid):

Let the error propagate, but map it: on-error-propagate that sets a stable error code and a 502/503 response rather than leaking the downstream's body.
Decide whether the Salesforce state is left partial, and if so, whether a compensation is needed, or whether the record should be written only after all prerequisite calls succeed.

The exam-level trap is assuming a caught error is a handled error - the choice of try boundary is the transaction boundary in Mule.