Besu Certified Professional: New Observer Node Peers Successfully but Never Leaves Block Zero

New Observer Node Peers Successfully but Never Leaves Block Zero

A private QBFT network with chain ID 2024 runs four validators and has produced about 2.3 million blocks at a 2-second block period. A reporting team needs a read-only node and provisions one on a new VM. An engineer copies a genesis file from an internal wiki page rather than the governance repository, then starts Besu with --network-id=2024 --p2p-host=10.20.4.17 --bootnodes=<validator-1 enode>. The node reports net_peerCount of 3 and admin_peers lists three validators, but eth_blockNumber returns 0x0 indefinitely and the data directory stays at a few megabytes. The log shows peers connecting and then disconnecting shortly afterwards, and repeating attempts to import the genesis block. The engineer's first instinct is to change --network-id to a different value to "force a resync", and to add --sync-mode=FULL because snap sync "looks stuck". The reporting team is asking why the node shows peers but no data.

1. Diagnose the root cause and describe the exact check that proves it, explaining what `--network-id` and chain ID each control.

Read the worked answer

Peers connecting and then dropping while the node stays at block zero is the classic signature of a genesis mismatch, not a networking problem.

The chain ID is a field inside the genesis file. It is part of the block-0 header and is what transaction signatures replay-protect against. Two nodes with different chain IDs have different block-0 hashes and cannot form one chain.
The network ID (--network-id) is a discovery/peer-filtering value only. It has no effect on block validation and changes nothing about whether blocks import. Changing it, as the engineer proposes, will not help and may additionally reduce the peer pool.

Proof. On the new node and on a validator, call eth_getBlockByNumber("0x0", false) and compare the returned hash (or compare the genesis file SHA-256 out of the governance repo). If the hashes differ, the node is on a different chain and will reject every block a validator sends. Disconnection codes and log lines naming a genesis or block-0 mismatch corroborate it.

Common causes, in practice: a wiki copy taken from a testnet or an earlier genesis revision; a changed alloc entry (for example a faucet balance); a different blockperiodseconds or requesttimeoutseconds; different qbft validator entries; or a missing transition. JSON whitespace and key ordering are irrelevant because the file is parsed, so reformatting is not a fix - any value difference is.

--sync-mode is irrelevant here: sync never starts because block 0 itself is rejected.

2. List the admission checks that should have run before this node was started, and what to do with the node now.

Read the worked answer

Checks before starting a node.

Genesis provenance: pull the genesis from the chain's governance repository at a tagged revision, not from a wiki or chat paste. Record its SHA-256 and the block-0 hash in the change ticket.
Identity fields: confirm chainId, the qbft validator list, blockperiodseconds, requesttimeoutseconds, and any alloc entries match the running network byte for byte.
Bootnodes and connectivity: enode URIs correct, TCP 30303 reachable in both directions, --p2p-host/--p2p-port set to what the firewall actually exposes.
Key handling: a reporting/observer node should not reuse a validator key; generate a fresh key for it, and never copy a validator's data directory to bootstrap it.
Role: run it without --miner-enabled, and expose only the RPC APIs the reporting team needs.

What to do now. Stop the node, delete the data directory it created (it contains only a rejected genesis, so there is nothing to preserve), replace the genesis with the authoritative file, verify the block-0 hash matches a validator's, then restart with snap sync (the default) rather than forcing full sync. Re-check net_peerCount and then watch eth_syncing or the head-height metric to confirm forward progress within a few blocks. Only after the block-0 hash matches should anyone look at sync performance.

3. The node must now catch up roughly 2.3 million blocks. Compare syncing from peers with copying a data directory from another node, including the risks of each.

Read the worked answer

Sync from peers (snap sync, default). The node downloads headers and block bodies and fetches state snapshots from peers. It is the safe default: no key material is shared, and the node verifies everything it receives. Risks: in a four-validator network the only peers serving state are the validators themselves, so a fresh sync adds real disk and network load to nodes that are also producing blocks; sync time is bounded by peer throughput and can be long; and if peers prune state or fall below the pivot, the sync can restart. Mitigate by syncing during a low-traffic window and watching validator disk I/O.

Copy a data directory. Much faster and avoids load on validators, but carries traps:

The key file in the data directory is the node's identity. Copying it duplicates the enode and, on an IBFT/QBFT node, the validator address. Two nodes with the same identity will be dropped as duplicates, and if the source was a validator, the copy is a second signer for the same address. Delete the copied key and let the node generate a new one, or supply a distinct --node-private-key-file.
Do not copy from a running node; stop the source or use a filesystem snapshot, or the database can be inconsistent.
The copied database must come from the same Besu release and storage format, otherwise the node migrates or refuses to open it.
Adjust --p2p-port and RPC ports so the two nodes can coexist on the same host.

Either way, the acceptance test is identical: block-0 hash matches and head height converges with the network, then the reporting workload is enabled.