SFTP Inbound Duplicates and Large-File Timeouts at a Logistics Hub
A logistics provider runs IBM Sterling B2B Integrator with two application nodes behind a load balancer and one perimeter server. Forty partners send EDI 214 and 990 files over SFTP. The inbound SFTP adapter polls every 30 seconds, and one business process deenvelopes, translates, and acknowledges each file. Files range from 2 KB to 900 MB; the largest partner sends a single 900 MB flat file each morning. In the past month operations reports three incidents where the 900 MB file was processed twice after the partner’s SFTP server dropped the connection mid-download, repeated “remote host closed connection” errors on files above 300 MB, and a morning backlog of 1,200 business process instances with some files waiting 40 minutes. The partner will not add a checksum, rename-on-complete convention, or split the file. The team may change adapter settings, business process logic, and scheduling, but not the partner’s behavior.
1. What is the most likely cause of the duplicate 900 MB processing, and what controls would you add without partner changes? Explain the order in which you would apply them.
Read the worked answer
The duplicate is almost certainly a non-atomic retrieve/delete/process sequence. The SFTP adapter can finish receiving the bytes, then the partner's server or the network drops the session before the adapter deletes or marks the remote file. On the next 30-second poll the same filename is visible, so it is retrieved again. A business process that has already committed the first copy will create a second order or acknowledgment unless something detects the redelivery.
Apply controls in this order:
- Prevent partial files from reaching the business process. Have the adapter or a staging script download to a local
.partfilename, then atomically rename to.readyonly after the transfer completes and the byte count is stable. Trigger the business process on.readyonly. This stops a half file from being translated, but it does not stop a complete file from being downloaded twice. - Add local idempotency. Compute a SHA-256 hash after staging, and persist a key of remote filename, size, modification time, and hash in a database table before processing. On redelivery, if the hash matches an existing successful key, quarantine the file and do not reprocess. If the filename matches but the hash differs, treat it as a new version and alert rather than silently overwriting history.
- Change cleanup semantics. If the adapter supports delete-after-business-process-success rather than delete-after-retrieve, enable it. If not, move the remote file to a processed directory only after the local commit; the partner refuses rename-on-complete, but the adapter may still be allowed to delete or move after success.
The hash adds I/O for a 900 MB file, but it is cheaper than duplicate orders. Do not rely on filename alone: partners sometimes resend corrected content under the same name.
2. Why do files above 300 MB fail with “remote host closed connection,” and what would you change? Consider that increasing a timeout alone may not help.
Read the worked answer
The error means the session ended before the adapter considered the transfer complete. It can come from the adapter's socket or session timeout, the partner's SFTP server idle or maximum-session limit, a firewall idle timer, or the adapter buffering too much data in memory. A fixed wall-clock limit on the partner side is also possible: if the partner closes every session after a set number of minutes, a slow 900 MB transfer will always fail regardless of our timeout.
I would first establish where the session is closed by correlating adapter logs, partner server logs, and a manual sftp transfer during the same window. If the partner closes after a fixed duration, increasing our timeout will not help. If our adapter closes first, raise the connection and session timeouts and the maximum file size above 900 MB, and verify that the adapter streams to disk rather than buffering the whole payload.
For a file this size I would also:
- give the large partner a dedicated adapter instance and polling schedule so its long session does not consume the same concurrency budget as small files;
- limit concurrent transfers per partner so two 900 MB sessions do not compete;
- verify JVM heap and temp space, because a 900 MB payload held in memory or copied twice can cause GC pauses and session drops;
- if the adapter supports restart/resume, use it only after confirming the partner supports append semantics, otherwise a resumed file can be corrupt.
Increasing timeouts is necessary only if the evidence points to our side; it is not a substitute for locating the closing party.
3. The 1,200-instance backlog occurs while the 900 MB file is processing. What scheduling and tuning changes would reduce the impact on small files?
Read the worked answer
The symptom suggests head-of-line blocking: one long-running transfer and its business process consume adapter sessions, worker threads, or database connections needed by small files. The fix is separation first, tuning second.
I would move the large partner to its own adapter, its own polling interval, and preferably an off-peak window, so a 40-minute transfer cannot sit in front of 2 KB files. If the partner insists on morning delivery, I would still isolate it with a dedicated adapter and a small concurrency limit, and route its output to a separate business process queue. Small-file partners keep the original adapter and a short poll interval.
Next I would collect evidence before changing thread counts: adapter session utilization, BP engine active threads versus configured maximum, database connection pool active/idle counts, and queue wait time. If the BP engine is at its thread ceiling but the database is not saturated, raising the BP thread limit modestly may help. If the pool is exhausted, raising threads will only increase wait time. I would also check whether the 997 is generated synchronously inside the same process; making acknowledgment generation asynchronous can shorten the critical path, but only if the partner accepts a delayed 997.
Finally, I would alert on per-partner queue age rather than only total backlog, so a single large file is visible as the cause before 1,200 small files accumulate.