Memory Exhaustion and Backend Latency in an App Connect Enterprise Flow

Two multipliers, both bad. Memory per in-flight message is set by the parsed tree, and the number of in-flight messages is set by concurrency and by how long each message is held.

Reduce memory per message:

• Full validation is expensive. Full validation against a message set creates additional objects and keeps validation artifacts around for each message, on documents that are 2 to 4 MB. If the payload is trusted (arriving from a controlled MQ queue with schema checks upstream), use content and value validation or turn full validation off in production and validate at the edge instead. Never validate a 4 MB document fully on a hot path unless the input is untrusted.
• Parse only what you use. The XMLNSC parser is lazy by default: accessing a single field should not build the entire tree. A retained large tree usually means something forces whole-document processing - a full validation, a Mapping node that navigates broadly, or a Compute node copying the entire tree with a generic copy before transformation. Restrict the transform to referenced fields.
• Stream the large payload out to object storage and pass a reference onward, so the MQ message itself is small.

Reduce messages in flight:

• Additional instances of the MQ input node multiply concurrent work. With two pods each holding several instances, each instance holding a 4 MB tree plus transform objects, a 1 GB limit is exhausted quickly. Cap instances deliberately rather than leaving defaults, and size the heap from measured per-message cost times intended concurrency.
• The 8 second p99 is the real driver. A message is held in memory for the entire backend call, so concurrency and duration multiply. With no explicit HTTP timeout, the default is minutes, so a stalled carrier holds every instance's message open until the timeout elapses. Set an explicit timeout comfortably above p99 but far below the default, and never leave it unset on an external call.

Little's Law gives the answer. Concurrency required equals throughput times latency: 300 messages per second times 8 seconds of backend latency equals 2,400 simultaneous in-flight requests. That is the number of threads - and therefore messages, parsed trees, and open HTTP connections - the flow must sustain just to keep the queue flat at the carrier's currently observed p99.

Why that is not achievable here. Two pods with a 1 GB container limit each give roughly 2 GB total, and each in-flight message carries a multi-megabyte tree. Even at a generous 50 KB of working state per message, 2,400 in flight needs hundreds of megabytes of live data per pod before JVM overhead, plus the threads to drive them. And 8 seconds is p99: during a carrier slowdown latency goes far higher, so the required concurrency rises exactly when memory is scarcest. That is the mechanism behind both symptoms - memory exhaustion and the 40,000-message backlog are the same event.

Therefore:

1. Adding pods is a linear fix to a multiplicative problem. It raises the ceiling, but the required concurrency still scales with latency, so you are buying time until the next slowdown. It is worth doing as a buffer, not as the answer.
2. Attack latency. Cache carrier reference data, move to an asynchronous or batched carrier interface if the carrier has one, or reduce payload size so the call is cheap. Halving latency halves required concurrency.
3. Decouple when the carrier is slow. If the carrier cannot be made faster, the flow should stop holding MQ messages open for the duration of an external call. A store-and-forward step, or an interface that accepts status updates asynchronously, converts an unbounded memory requirement into a bounded queue with visible backlog.
4. Apply backpressure deliberately. Cap additional instances so the queue grows instead of the heap. A deep queue is recoverable; a restarting integration server with in-flight non-idempotent POSTs is not.

The consequence is duplicate status records and, potentially, wrong shipment state. If a pod dies after the carrier accepted the POST but before MQ committed the transaction, the message rolls back and is redelivered. The flow POSTs again, and the carrier creates a second status record. With retries, timeouts and restarts all in play, the same shipment can be posted several times. Because the endpoint is not idempotent, the duplicate is not harmless - downstream consumers may see a shipment apparently regress or regress-and-advance, which is exactly the kind of corruption that is hard to unpick later.

Making it safe:

1. Establish a durable business key. Generate or derive a deterministic identifier for the status event (shipment ID plus event sequence or event timestamp) and carry it in the message body or a header, not only in the transport.
2. Make the carrier deduplicate. The strongest fix is an idempotency key that the carrier honours: same key, same effect, regardless of repetition. If the carrier supports it, use it; if it does not, ask - this is a contract issue, not a code issue.
3. If the carrier cannot deduplicate, maintain a local record of committed event IDs, written in the same transaction that commits the MQ message where possible, and check it before posting. This narrows the duplicate window but cannot close it entirely, because the crash can occur between the carrier's response and your commit. Say so explicitly rather than claiming the problem is solved.
4. Handle the timeout path separately. A timeout is not a failure; the carrier may have succeeded. Retrying a timed-out non-idempotent POST is the classic duplicate generator. On timeout, reconcile by querying the carrier for the event before re-posting, and route unreconciled cases to a backout queue with an owner and alert rather than blind retries.
5. Make the failure visible. Alert on backout queue depth and on the reconciliation queue, so a carrier incident produces a known work item rather than silent duplication.