Skip to main content

Validator Operations

Day-two work: knowing your node is healthy, keeping the one secret you cannot regenerate, and recognising the failures that matter.

Telemetry​

Your node reports to telemetry.orbinum.network by default — block height, finalized height, peer count, transaction pool, version and approximate location.

The Compose file already sets this. Three parts of the value are load-bearing:

TELEMETRY_URL=--telemetry-url "wss://telemetry.orbinum.io/submit/ 1"
  • The quotes stay. The node parses "<url> <level>" as a single argument. Without them the level is read as a separate flag and startup fails.
  • The trailing slash stays. /submit without it does not upgrade.
  • 1 is the verbosity level, and for a validator it is not optional: level 1 sends afg.authority_set, the only message carrying your validator address. That is what populates the Address column. RPC nodes stay at 0 — they have no address to report.

To opt out entirely, set the variable to empty and recreate the container. Telemetry is an outbound connection, so it opens no port and works on a node whose RPC is loopback-only.

Changing telemetry needs a recreate

A container's command line is fixed at creation. docker compose restart brings the node back with its old arguments:

docker compose up -d --force-recreate orbinum-validator

Metrics and alerts​

Prometheus metrics are exposed on 9615, bound by METRICS_BIND — keep it on a private interface.

The four alerts worth having, in Prometheus rule form:

groups:
  - name: orbinum-validator
    interval: 30s
    rules:
      - alert: NodeOffline
        expr: up{job="orbinum-node"} == 0
        for: 2m
        labels: { severity: critical }

      - alert: LowPeerCount
        expr: substrate_sub_libp2p_peers_count < 3
        for: 5m
        labels: { severity: warning }

      - alert: FinalityStalled
        expr: |
          (substrate_block_height{status="best"} -
           substrate_block_height{status="finalized"}) > 100
        for: 5m
        labels: { severity: critical }

      - alert: HighMemoryUsage
        expr: |
          (process_resident_memory_bytes /
           node_memory_MemTotal_bytes) > 0.9
        for: 10m
        labels: { severity: warning }

FinalityStalled is the one to act on first. GRANDPA needs more than two thirds of the set voting; with a small validator set, a couple of nodes offline stops finality for everyone while block production continues. A growing gap between best and finalized is the signal.

Back up your keystore​

Chain data lives in a named Docker volume and can always be resynced. The keystore cannot. It holds the private halves of your session keys.

# Copy the keystore out of the container, then encrypt it
docker cp orbinum-validator:/data/chains/orbinum_testnet/keystore ./keystore-backup
tar -czf keystore-$(date +%Y%m%d).tar.gz keystore-backup
gpg --symmetric --cipher-algo AES256 keystore-*.tar.gz
rm -rf keystore-backup
Never run two nodes with the same keys

Restoring this backup onto a second machine while the first is still running makes both sign with the same identity. On Orbinum that costs you nothing today — there is no slashing — but it produces conflicting votes that damage finality for the whole network, and it is exactly what slashing will punish once it exists.

If you are migrating, stop the old node first and confirm it is stopped.

Losing the keystore is recoverable: generate new session keys with author_rotateKeys and submit session.setKeys again. You keep your slot; you just miss blocks until the next session.

Updates​

Watchtower ships in the Compose stack and polls every 5 minutes, rolling-restarting the node when a new image is published. Nothing to do.

To force an update immediately:

cd node-deploy/testnet/validator
docker compose pull && docker compose up -d

Check what Watchtower has done:

docker compose logs orbinum-watchtower

Runtime upgrades apply through on-chain governance and need no operator action — the node adopts the new runtime when the upgrade block finalizes.

Troubleshooting​

The node has no peers​

Almost always one of two things:

  1. Port 30333 is not reachable from the internet. Open in ufw is not enough — check your provider's security group, and any NAT in front of the host. Test from elsewhere: nc -z <your-ip> 30333.
  2. RESERVED_NODES is set. A non-empty value activates --reserved-only, which makes the node refuse every peer not on the list — the bootnodes included. Public validators leave it empty. See step 2.

The node is synced but never authors​

Work through these in order:

CheckHowExpected
Session keys are on-chainsession.nextKeys(yourAccount) in Chain stateReturns your key blob
You are in the approved setvalidatorSet.approvedValidatorsContains your account
You are in the active setsession.validatorsContains your account
Enough time has passed—2 session boundaries ≈ 2 h since addValidator

If approvedValidators contains you but session.validators does not, and more than two hours have passed, the usual cause is that the keystore no longer holds the keys registered on-chain — you rotated keys after setKeys, or restored the container without the volume. Re-run author_rotateKeys and submit setKeys again.

Finality is stalled​

Check the best-versus-finalized gap on telemetry. If block production continues but finality does not advance, the network is short of GRANDPA votes — not something one operator fixes alone. Keep your node up and reachable, and tell the team.

The container will not start​

docker compose logs --tail 100 orbinum-validator

Two common causes: port 30333 already in use on the host, or the chain-spec mount path is wrong after moving the checkout. Both name themselves clearly in the logs.

Wiping chain data is a last resort

docker compose down -v deletes the volume, forcing a full resync — hours of it. It also deletes the keystore unless you have a backup. Exhaust everything else first.

Next Steps​