Besu Certified Professional: Partner Node Onboarding Fails Despite a Correct Allowlist Entry

Partner Node Onboarding Fails Despite a Correct Allowlist Entry

A five-node permissioned network uses onchain permissioning: an accounts allowlist contract and a nodes allowlist contract, with node permissioning enabled on every node. A partner team is onboarded with a read-only node, rpc-2. To save the partner three days of sync time, an engineer copies rpc-1's entire data directory - including the key file - onto the new VM, changes the ports, starts Besu with --p2p-host set to the partner's public IP, and submits an admin transaction adding the resulting enode to the nodes allowlist. The transaction succeeds. rpc-2 shows zero peers, rpc-1's peer count drops from four to two and then recovers, and validator logs show connections from a node ID that is already connected. Separately, the partner's deployer account was added to the accounts allowlist in block 900,120 and the receipt status is 0x1, yet the partner's first contract-deployment transaction is rejected with an account-not-allowlisted error.

1. Explain why `rpc-2` has no peers and why `rpc-1`'s peer count dropped, and describe the remediation.

Read the worked answer

The copied key file is the node's identity: it derives both the P2P enode and, on IBFT/QBFT chains, the validator/account address used for signing. By cloning the data directory, the engineer created a second node claiming rpc-1's identity.

What follows is exactly what the logs show. Peers that are already connected to rpc-1 see a second connection presenting the same node ID and drop it as a duplicate or self-connection; some peers may displace the original connection, which is why rpc-1's count fell to two and then recovered as peers reconnected. Adding the enode to the allowlist was correct but useless - the contract already contained that enode, because it belongs to rpc-1. Node permissioning accepted the entry and the P2P layer rejected the duplicate.

The risk is not only connectivity: two nodes signing with the same key is a duplicate signer. If the source node was a validator, both instances could sign rounds for the same address, and monitoring, peer scoring, and any per-node accounting become meaningless.

Remediation:

Stop rpc-2. Delete the cloned data directory - do not simply delete key and keep the rest, because the database also carries the copied identity metadata and any validator state.
Start rpc-2 fresh with the correct genesis and a newly generated key (or an explicit --node-private-key-file under the partner's control), and confirm it is not a validator address.
Read the new enode from admin_nodeInfo, submit a new allowlist transaction, wait for the block to be imported network-wide, then confirm net_peerCount rises.
Record the rule: never distribute a data directory for bootstrap; distribute a filesystem snapshot with the key removed, or sync from peers.

2. The allowlist admin key is currently pasted into a startup shell script on a validator. Describe the key-management and RPC-exposure changes you would require before the next partner is onboarded.

Read the worked answer

The admin key is the highest-value secret in the system: whoever holds it can add or remove accounts and nodes. A key in a startup script is in shell history, process listings, backups, and possibly the repository.

Key management:

Move the admin key out of the validator entirely. Administration should be a separate offline or jump-host operation using an account whose key is held in a keystore with a strong passphrase, or in an HSM/KMS-backed signer, and never in an environment variable on a block-producing node.
Separate roles: the validator key signs consensus messages; the permissioning admin key signs allowlist changes. They must be different keys held by different people under different controls.
Set an explicit rule for rotation and revocation, and rehearse removing a partner node and account - an allowlist you cannot safely prune is not a control.
Do not bootstrap nodes by copying data directories; the key file inside is identity material, as this incident showed.

RPC exposure:

On internet-facing endpoints, enable only ETH, NET, and WEB3. Do not enable ADMIN, DEBUG, TXPOOL, or MINER there - those leak peer and account data or allow disruptive operations.
Bind RPC to internal interfaces, enforce --host-allowlist, and terminate TLS at the node or at a proxy that enforces mutual TLS for partner access.
Enable RPC authentication (credentials file, or JWT where the API supports it) so that an open port is not an open door, and keep an administration path that is reachable only from the management network.
Log and alert on allowlist contract writes; an unexpected admin transaction should page someone, not appear in a weekly report.