Reconciliation Functions: 504s, duplicate files, and a nine-hour drain

The 230-second failure is the platform's fixed HTTP response timeout on the Consumption plan, not functionTimeout. Raising functionTimeout (max 10 minutes on Consumption) will not fix a front end that hangs waiting on a response, so the shape of the call has to change.

Recommended: Durable Functions async HTTP API pattern.

1. The HTTP trigger validates the request, calls StartNewAsync on an orchestrator, and returns 202 Accepted with the statusQueryGetUri from CreateCheckStatusResponse.
2. The orchestrator calls an activity that invokes the synchronous library. The library call must live in the activity, never in the orchestrator, because orchestrator code must be deterministic and replay-safe.
3. Add RetryOptions on the activity (the library's 3-8 minute runtime is exactly the kind of transient fault that benefits from a bounded retry), and set MaxOrchestrationHistorySize awareness by keeping activity payloads small - pass a blob path or message id, not the batch itself.
4. The caller polls the status endpoint, which also survives function app restarts and scale-in.

Cheaper alternative if no orchestration is wanted: enqueue a start message and return 202 with a run id; the worker writes a status row to Table/Cosmos storage that the caller polls. You reimplement status tracking, retries, and timeouts yourself - Durable gives them for free.

Either way, do not solve this by making the HTTP call asynchronous in the client only; the function still times out and the work still dies with it.

Blob triggers are at-least-once, not exactly-once. The runtime stores receipts in the azure-webjobs-hosts control container in AzureWebJobsStorage, but a host restart, scale-out race, or lease/visibility expiry re-delivers the same blob. Two independent causes are plausible here and you should check both:

• The carrier genuinely delivered two files with different names but the same content.
• The Service Bus path re-delivered, not the blob path: with the default 5-minute message lock and a 3-8 minute library call, the lock expires, Service Bus redelivers the message, and the batch is processed twice. This is the more likely culprit given the runtimes and the queue backlog, and it also explains why duplicates cluster on slow nights.

Fixes, in order of value:

1. Make the handler idempotent rather than trying to prevent redelivery. Derive a deterministic key from (carrier, fileName, content hash) and attempt AddEntity (Table) or CreateItem (Cosmos) with that key; a 409/conflict means already done, so exit cleanly. On the blob side, write a completion marker with If-None-Match: *.
2. Raise maxAutoLockRenewalDuration past the worst-case library runtime, or shorten the unit of work by splitting batches so each message finishes well inside the lock.
3. Emit the business key (batch id, file hash) as a custom dimension on every log and telemetry item so duplicates are grep-able rather than discovered by finance.

Do the arithmetic before changing the plan. 40,000 messages across 20 instances over four hours (14,400 s) is 2,000 messages per instance, or about 7.2 seconds each. A batch that takes 3-8 minutes means you cannot meet the window at concurrency 1 - you need roughly 40-70 messages in flight per instance, or you need to shrink the unit of work.

Levers, cheapest first:

1. Check host.json. maxConcurrentCalls (default 16 for Service Bus) and prefetchCount are the direct throughput dials. Also check whether the queue or topic is partitioned - a non-partitioned queue has a single message store and will not fan out to 20 instances efficiently.
2. Batch inside the function. Pull several settlement messages and call the library once per batch rather than per message. This is where most of the win is, because the 3-8 minute cost is per call, not per line item.
3. Stop the redelivery tax. If locks are expiring, a large share of those 40,000 deliveries is wasted work; fixing the lock settings may alone remove hours.
4. Only then change plan. The 20-instance ceiling on Windows Consumption is a platform limit. Elastic Premium removes the Windows Consumption limits, gives more cores per instance, and removes the 10-minute cap - but it is a monthly floor cost, so justify it with the numbers above, not as a first move.

Deliberately not recommended: switching to an App Service plan for a bursty nightly job, or parallelising the library with threads, which fights the Consumption plan's single-core-per-instance reality.