Skip to main content

Troubleshooting

Ordered by how often they bite. The first two cover most incidents.

The node has no peers​

Almost always one of two things:

  1. Port 30333 is not reachable from the internet. Open in ufw is not enough — check your provider's security group, and any NAT in front of the host. Test from elsewhere:

    nc -zv <your-public-ip> 30333

  2. RESERVED_NODES is set. A non-empty value activates --reserved-only, which makes the node refuse every peer not on the list — the bootnodes included. A public node leaves it empty.

Check the current count:

docker exec orbinum-validator curl -s -H 'Content-Type: application/json' \
  -d '{"id":1,"jsonrpc":"2.0","method":"system_health"}' \
  http://localhost:9944
The indexer archive is meant to have no inbound peers

It deliberately sets no --public-addr and takes no inbound P2P — outbound connections still get their replies. Zero inbound peers is correct there.

The node is synced but never authors​

Validators only. Work through these in order:

CheckHowExpected
Session keys are on-chainsession.nextKeys(yourAccount) in Chain stateReturns your key blob
Keystore matches the chainauthor_hasSessionKeys with that blobtrue
You are in the approved setvalidatorSet.approvedValidatorsContains your account
You are in the active setsession.validatorsContains your account
Enough time has passed—2 session boundaries ≈ 2 h since addValidator

The second row catches what the others miss. The chain can hold a perfectly valid key blob for your account while your node holds none of the matching private halves — that happens if you ran author_rotateKeys on a different machine, or recreated the container without its volume. Every other check passes and the node still never authors:

docker exec orbinum-validator curl -s -H 'Content-Type: application/json' \
  -d '{"id":1,"jsonrpc":"2.0","method":"author_hasSessionKeys","params":["<your 128-hex blob>"]}' \
  http://localhost:9944

false — rotate again on this node and submit session.setKeys with the new blob. You keep your slot in approvedValidators; the set is re-read from session.nextKeys at every rotation, so nothing needs re-approving.

true, you are in session.validators, and still no blocks — that is a different problem. Check the node is not stuck syncing, and tell the team.

session.purge_keys is permissionless

An approved validator that purges its keys keeps its slot in the schedule and produces nothing. The key check runs at every session rotation, not just when you were added.

Finality is stalled​

Check the best-versus-finalized gap on telemetry. If block production continues but finality does not advance, the network is short of GRANDPA votes — not something one operator fixes alone. Keep your node up and reachable, and tell the team.

The container will not start​

docker compose logs --tail 100

Two causes account for most of it, and both name themselves in the logs:

  • Address already in use — something else holds 30333 on the host. Find it with sudo ss -ltnp | grep 30333.
  • Chain-spec mount error — testnet-spec.json is not where the Compose file expects it, usually after moving the checkout.

After editing .env or the Compose file:

docker compose up -d --force-recreate

High memory usage​

RPC and indexer nodes are capped in their Compose files so a query flood cannot starve Caddy and the OS. Defaults: RPC RPC_MEM_LIMIT=6g / RPC_CPUS=1.5, indexer 4g / 1.5. The validator has no caps.

docker stats --no-stream

Tune in .env and recreate. Leave roughly 2 GB and a core for the OS and Caddy.

Wiping chain data is a last resort

docker compose down -v deletes the volume, forcing a full resync — hours of it. On a validator it also deletes the keystore unless you have a backup. Exhaust everything else first.