Settlement dApp Suffers Reverts and Failed Historical Queries

eth_estimateGas is a simulation against a specific state - latest unless told otherwise - and the result is only valid for that state. The signed transaction is executed later, at a different state.

Three effects break the 1.0 multiplier:

1. State drift between estimation and inclusion. Another transaction can move the contract's state (a token balance, an allowance, a liquidity pool ratio, a batch counter) so the mined execution takes a different path with more storage writes, more external calls, or an outright revert. A revert is not a gas-sizing problem at all - raising the limit just makes it fail more expensively.
2. Pending transactions are invisible to latest. If the sender has an earlier pending transaction that changes a nonce or a state the batch depends on, estimation runs against state that will not exist when the transaction is mined.
3. Nested call gas forwarding. The batch calls several token contracts. Because of the 63/64 forwarding rule, an inner call can be starved, and any single out-of-gas in an inner frame reverts the whole batch. Contracts that branch on gasleft() also need headroom.

Strategy: estimate on pending when the submission path allows it, add a margin (20–30% is typical, not 0%), and - more importantly - simulate with eth_call at the exact block and state the transaction will execute against, using state overrides if needed to pin the inputs. Treat estimation failure as a signal to fix the payload, not to raise the limit: check the revert reason via eth_call and the receipt of the failed transaction before resubmitting. Finally, on out-of-gas, retry with the same nonce and a larger limit rather than submitting a new nonce, or the pending-nonce complaints continue.

Besu's default Bonsai configuration prunes historical state. The node keeps enough recent state to serve latest and a short window of recent blocks, but state for blocks from six months ago has been pruned, so a historical eth_call cannot resolve the account or storage slots it needs and fails with a missing-trie-node style error. Increasing heap does nothing; the data is not on disk.

Options, in order of preference:

1. Dedicated archive node. Run a separate node with full sync and pruning disabled, so all historical state is retained. Point only the audit job at it. Sizing is the constraint - full state on a busy chain is several times the pruned footprint - so give it its own disk and do not put it behind the production load balancer.
2. Retain what is needed. If the audit only needs a handful of accounts, capture the data at request time going forward (store the eth_call results, or snapshot the relevant storage proofs) instead of relying on replay months later. This fails for look-back requests already in scope.
3. Query an external indexer that has retained historical state, accepting the trust shift.

Do not solve this by disabling pruning on a production validator: it inflates disk and can affect block-production latency. Also confirm that the archive node's genesis and chain ID match exactly, and that the audit path uses an explicit block number rather than latest - the audit's value depends on reproducible block pins, and latest silently changes the answer.

eth_getLogs from block 0 to latest over roughly 2.3 million blocks is not a query - it is a full-chain scan executed every 30 seconds, materializing every matching log in memory before serializing the response. With a 16 GB heap, two overlapping scans are enough to trigger the OOM killer. The restart then makes the next poll scan an even larger range, so the failure feeds itself.

Fix:

1. Bound the range. Query only the blocks produced since the last successful poll (30 seconds at a 2-second block period is roughly 15 blocks), with a small overlap and deduplication by (blockHash, logIndex). Persist the last processed block so a restart resumes rather than rescans.
2. Respect the node's cap. Besu enforces a maximum block range for log queries; a request beyond it is rejected. Size the monitoring window well below the configured cap and alert if the gap ever approaches it.
3. Prefer filters or subscriptions. Use eth_newFilter + eth_getFilterChanges for polling or eth_subscribe over WebSocket for push, so the node maintains the cursor instead of rescanning.
4. Move indexing off the producers. Use a dedicated indexer/RPC tier, and if it must be Besu, put it behind the load balancer with the production nodes protected by connection and request limits.
5. Set memory deliberately. Size the heap to the largest legitimate response, and monitor per-request memory, not just process liveness. Raising the heap alone only postpones the same failure.