Nightly Batch Bottleneck: When to Chunk vs. Parallelize
A Boomi process ingests 1.2 million order records nightly from an on-premise Oracle database via a Database Connector. The process uses a single Get operation with no paging, followed by a Map shape and a 5-way branch that sends records to three different REST endpoints and two internal queues. Runtime has grown from 45 minutes to 6.5 hours over 18 months. The database team reports that the connection holds a single long-running transaction for over 4 hours, blocking index maintenance. The target endpoints accept batches of up to 500 records. Peak memory on the Boomi Atom is 6 GB, and the JVM heap is set to 4 GB. The business requires the batch to complete within 2 hours to meet the 6 AM cutover. What design changes would you evaluate, and how would you sequence them?
1. Identify the primary architectural cause of the runtime degradation and explain why increasing the Atom heap alone will not solve it.
Read the worked answer
The primary cause is unbounded document accumulation: a single Get operation pulls the entire result set into memory before any downstream shape runs, so the Atom holds 1.2M records in the process's document stream. Even if heap were doubled, the process would still serialize the entire result set through a single thread and hold a long transaction. The 5-way branch exacerbates this because each branch iterates the full stream, multiplying memory pressure and outbound calls. Increasing heap only delays OutOfMemoryError; it does not reduce transaction duration, improve throughput, or address the blocking index maintenance. The correct fix is to introduce streaming/paging at the source and batch-oriented processing downstream.
2. Propose a redesigned process flow that meets the 2-hour SLA, including specific configuration choices and trade-offs.
Read the worked answer
Redesign around paged reads and batched writes. 1) Replace the single Get with a paged Get using a keyset or rownum-based pagination of 5,000 records per page, driven by a loop or a chunked batch. 2) Insert a Data Process shape to split each page into batches of 500 (the endpoint limit) before the branch. 3) Convert the 5-way branch to a parallel processing pattern only if the endpoints are idempotent and can handle concurrency; otherwise use sequential batch calls with retry and backoff. 4) Move the two internal queue writes to asynchronous JMS or a queue connector so they do not block the REST calls. 5) Set the Atom's document cache and commit settings to release memory per batch. Trade-offs: paging increases database round trips but shortens transaction duration; parallel branches improve wall-clock time but require idempotency keys and a dead-letter path. With 500-record batches, 1.2M records require 2,400 calls; at 20 concurrent calls and 200 ms average latency, the REST portion completes in roughly 24 seconds of compute, leaving ample margin for mapping and queue writes within 2 hours.