Skip to main content

Troubleshooting

Ordered by how often they bite. The first two cover most incidents.

The node has no peers​

Almost always one of two things:

  1. Port 30333 is not reachable from the internet. Open in ufw is not enough — check your provider's security group, and any NAT in front of the host. Test from elsewhere:

    nc -zv <your-public-ip> 30333

  2. RESERVED_NODES is set. A non-empty value refuses every peer not on the list, bootnodes included — see Peering. A public node leaves it empty.

Check the current count with the system_health call in Health checks.

The node is synced but never authors​

Validators only. Work through these in order:

CheckHowExpected
Session keys are on-chainsession.nextKeys(yourAccount) in Chain stateReturns your key blob
Keystore matches the chainauthor_hasSessionKeys with that blobtrue
You are on the allowlist (permissioned admission)validatorSet.approvedValidatorsContains your account
You are a candidatestaking.validators(yourAccount)Returns your commission
You were electedsession.validatorsContains your account
Enough time has passed—Candidacies count at the next election, about 2 h before the era ends; the set takes over at the next era
Your node meets the minimum versionvalidatorSet.minAuthorVersion against orbinum-node --versionNone, or at or below your version

The second row catches what the others miss — see Confirm the keystore matches the chain for the call and what to do on false.

A node below the minimum version says so in its log — at startup, and again at each slot it skips:

this node's version 0.5.0 is below the minimum 0.6.0: it cannot author until it upgrades

A binary older than 0.4.0 declares no version at all and logs block author declared no node version; minimum is … instead. Either way it keeps importing and voting; upgrading and restarting it is the whole fix — see Minimum node version.

true, you are in session.validators, the version is fine, and still no blocks — that is a different problem. Check the node is not stuck syncing, and tell the team.

Keys and the active set

A validator in the current or queued set cannot change or purge its keys (code 8). To rotate them, stop validating first — see Rotate session keys.

Finality is stalled​

Check the best-versus-finalized gap on telemetry, or the FinalityStalled alert. GRANDPA needs more than two thirds of the set voting; with a small set, a couple of nodes offline stops finality for everyone while block production continues — so the chain looks alive from the outside, and the failure otherwise goes unnoticed for hours. It is not something one operator fixes alone. Keep your node up and reachable, and tell the team.

The node rejects blocks the network accepted​

The log repeats Digest item must match that calculated on import, and the node falls behind or splits off on its own fork. Check the binary first (<node> is orbinum-validator or orbinum-rpc-node):

docker exec <node> orbinum-node --version

Nodes before 0.3.1 recorded storage proofs while importing but not while authoring, so they metered EVM proof-size gas differently from the block's author, computed a different Ethereum block, and rejected it. A block that calls the same large contract more than once is enough to trigger it. Update to the current release — see Image updates.

Merkle proof requests are refused as busy​

privacy_getMerkleProof, privacy_getMerkleProofByCommitment and privacy_getSubtreeRoots run on at most half the node's cores at once. Further requests queue — up to 128 of them, for up to 2 seconds each — and only past that are they refused with error -32009, Merkle proof queue is full, try again later.

Seeing it means the node is saturated or one client is flooding it. Retry with backoff; on a public RPC, check the rate limits in front of the node — see Public RPC Node.

The container will not start​

docker compose logs --tail 100

Two causes account for most of it, and both name themselves in the logs:

  • Address already in use — something else holds 30333 on the host. Find it with sudo ss -ltnp | grep 30333.
  • Chain-spec mount error — testnet-spec.json is not where the Compose file expects it, usually after moving the checkout.

A fix to .env or the Compose file only takes effect after a recreate.

High memory usage​

The public RPC is capped in its Compose file so a query flood cannot starve Caddy and the OS. Defaults: RPC_MEM_LIMIT=6g / RPC_CPUS=1.5. The validator has no caps.

docker stats --no-stream

Tune in .env and recreate. Leave roughly 2 GB and a core for the OS and Caddy.

Wiping chain data is a last resort

docker compose down -v deletes the volume, forcing a full resync — hours of it. On a validator it also deletes the keystore unless you have a backup. Exhaust everything else first.