Four-Validator IBFT 2.0 Network Stalls During a Parallel Patching Window

IBFT 2.0 commits a block only when it collects commit messages from a quorum of 2F + 1 validators, where F = floor((N - 1) / 3).

• N = 4 → F = floor(3/3) = 1 → quorum = 3 of 4.

With v3 and v4 rebooting, only two commit messages could ever be collected, which is below the quorum of three. The remaining validators kept proposing and entering round changes, but no block could be committed. That is exactly what the logs show: new rounds starting, commit counts stuck at two. The team's reasoning applied a majority test (2 of 4 would need 3 anyway, and majority is the wrong threshold here); the correct test is the BFT quorum, which is strictly larger than a simple majority for small N.

Crash-fault tolerance for liveness is N − (2F + 1):

• N = 4 → 1 validator offline at a time.
• N = 5 or 6 → F = 1, quorum = 3, so 2 (N=5) or 3 (N=6) may be offline.
• N = 7 → F = 2, quorum = 5, so 2 may be offline.

So a five-validator set would have survived this window with a quorum of three, and would have tolerated two simultaneous outages. Note the cost: more validators means more message traffic per round and a longer round-trip budget, so requesttimeoutseconds must remain comfortably above worst-case network latency.

The transactions were accepted into each receiving node's transaction pool and gossiped, but could not be included because no block was produced. They are not lost by default; they stay pending and are mined in nonce order once blocks resume.

Before asking for resubmission, verify in this order:

1. Presence: query eth_getTransactionByHash for a sample, and txpool_besuPendingTransactions / txpool_besuStatistics on v1 and v2 to see whether the transactions are still pooled or were evicted.
2. Eviction: if the pool exceeded its configured maximum (default in current releases is 4,096 transactions) or the transactions were underpriced relative to a gas-price spike, the lowest-priced entries are dropped. Only those need resubmission.
3. Nonce continuity: use eth_getTransactionCount(address, "pending") per sender. A dropped middle transaction leaves a nonce gap, so later transactions can never be mined until the gap is filled - resubmitting the tail without filling the gap simply recreates the stall.
4. Duplicates: if the partner already resubmitted the same nonce to a different node, both copies are in the pool; only one will be mined, but the partner must reconcile receipts by hash rather than assuming failure.

Also log which RPC endpoint each transaction arrived on. When v3 and v4 rejoined, they re-gossiped nothing, since they were down - pending transactions created during the freeze exist only in the pools of nodes that were online.

Durable fix. Grow the validator set so that one maintenance action never removes the quorum. Moving from 4 to 5 validators raises the quorum to 3 and tolerates two simultaneous outages; 6 validators tolerates three. Do not over-provision blindly: each added validator adds two message rounds per block, so re-baseline requesttimeoutseconds against measured p99 latency after the change, and confirm the extra VMs are in a failure domain independent of the existing ones.

Changing the set. In IBFT 2.0 the validator set is not edited by hand in genesis on a running chain. Sync the new node fully, confirm its validator address matches the key you intend to use, then submit ibft_proposeValidatorVote from a validator with the new address. The vote takes effect when a majority of validators accept it; verify with ibft_getValidatorsByBlockNumber("latest") and confirm the change is visible on every node before retiring anything.

Patching runbook.

1. Pre-check: count live validators and confirm live − 1 ≥ quorum. Abort if not.
2. Patch exactly one validator. Before starting the next, gate on: head height equal to the network head, the node participating in rounds, and a successful block signed by it.
3. Alert on head-height divergence and on round-change rate, not just on process liveness - the process was healthy during this incident.
4. Record the maintenance window in the runbook with an explicit no parallel reboots rule, and make the quorum pre-check a required, signed step rather than a judgement call.