Nightly 45 MB partner order file fails with OutOfMemoryError
OrderLink ingests a nightly pipe-delimited purchase-order file from 42 retail partners. The file averages 45 MB and 260,000 lines, peaking at 380,000. The flow is FTP Adapter (download) → Stage File (read) → mapper → REST Adapter (POST /orders/v2) on the partner API, which throttles at 40 requests/second and rejects bodies over 1 MB. Last week the flow began failing at 9–12 minutes with OutOfMemoryError and 'XSLT transformation timed out'; at 40,000 lines it used to finish in 6 minutes. The mapper tokenizes the entire file body, sorts line items globally, and builds one JSON document. The SLA is 30 minutes and a rerun must not duplicate orders. A developer proposes setting for-each Degree of Parallelism to 16 and raising the invoke timeout to 300 seconds.
1. Where is the memory pressure actually coming from, and which design change addresses it first?
Read the worked answer
The pressure is created by materializing the whole 45 MB document three times: as the Stage File read result, as an XSLT DOM when tokenize() splits the body, and as the aggregated JSON. DOM expansion is commonly 5–10×, so a 380,000-line peak file can need several hundred MB of heap - which is exactly why 40,000 lines worked and 260,000 does not.
First change: stop mapping the file as a single document. Read the file using the Stage File 'read file in segments' option (or a delimited nXSD with segment reads) so each unit of work is a bounded subset, wrap mapper plus REST invoke in a for-each with a batch size of 100–200, and emit one order object per iteration, each well under the 1 MB limit. Drop the global sort of line items - sort in the source extract or let the target order them - because a global sort forces buffering of everything before the first write. Each transform then operates on a small constant-size fragment, which fixes both the heap error and the XSLT timeout.
2. A colleague wants Degree of Parallelism 16 plus a 300-second invoke timeout to meet the SLA. Evaluate that proposal.
Read the worked answer
Reject it as written, although low-degree parallelism is legitimate. Each parallel thread holds its own copy of the current segment, the invoke context and an open HTTP connection; at 16 threads peak heap can exceed the single-threaded version that already fails, so OutOfMemoryError may become more frequent rather than rarer. Throughput is also capped by the partner's 40 requests/second limit - 16 threads mainly queue up and provoke 429s, which this flow does not currently back off from.
Recommend 2–4 threads, batch size 100–200, explicit throttling to stay under the documented rate, and a 429 handler with exponential backoff plus jitter so the whole flow does not retry in lockstep. Keep the invoke timeout close to the API's stated response SLA (roughly 20–30 s), not 300 s: a long timeout converts fast failures into hangs and multiplies the concurrency held by stuck threads, which is worse for both heap and SLA than failing quickly.
3. How do you prove the redesign before it touches the nightly production schedule?
Read the worked answer
Replay a synthetic file at the peak profile in a non-production instance of the same shape: 380,000 lines, wide rows, and some orders with 200+ line items, since line-item fan-out drives the JSON size. Measure peak heap on the runtime and, if any leg runs through a connectivity agent, on the agent JVM; records per second at the target; 429 rate; and total elapsed time against the 30-minute SLA.
Test idempotency separately: a mid-file failure followed by a partial rerun is precisely when duplicates appear, so verify the target honors an Idempotency-Key or upserts on order id. Then confirm exactly-once segment processing by comparing source line count with target upsert count - skipped first/last records and off-by-one segment boundaries are the classic defects in this pattern. Only after those numbers are stable should the new version be activated, and the old version must be deactivated in the same window so two schedules never overlap.