Node Operations

Orbinum is built on Substrate with Frontier for EVM compatibility. This means node operation follows standard Substrate patterns — if you've run a Polkadot, Kusama, or any Substrate-based node, the concepts are familiar.

This guide covers Orbinum-specific configuration and operational considerations beyond standard Substrate node management.

The Orbinum testnet runs from a pre-built image on GitHub Container Registry (ghcr.io/orbinum/node:testnet-latest). Operators pull that image with Docker Compose — they do not compile from source. The compose stacks live in docker/testnet/ in the node repository.

For Complete Setup Instructions

See the Running a Node guide for the quick start, and the node repo's docs/rpc-node-setup.md for the full public-RPC walkthrough.

---

Node Types

Orbinum supports three standard Substrate node types, each serving different network roles.

Full Node

Validates all blocks and stores current state. Does not participate in consensus.

Requirements:
4+ cores | 16 GB RAM | 500 GB SSD

Archive Node

Maintains complete historical state for every block since genesis. Required for explorers and analytics.

Requirements:
8+ cores | 32 GB RAM | 2+ TB NVMe

Validator Node

Participates in consensus (Aura + GRANDPA) by producing blocks and finalizing chains. Requires a 1,000 ORB bond and governance approval.

Requirements:
8+ cores | 32 GB RAM | 1 TB NVMe | 99%+ uptime
Bond: 1,000 ORB (locked on-chain)

---

Orbinum-Specific Configuration

Verification Keys (On-chain Registry)

Unlike standard Substrate setups that rely on static key assumptions, Orbinum manages ZK circuit verification keys through an on-chain versioned registry in pallet-zk-verifier.

Verification key lifecycle:

• Generated from circuit compilation in orbinum/circuits
• Registered on-chain by (circuit_id, version) via governance/sudo
• Activated per circuit with setActiveVersion
• Verifier reads effective VK from on-chain storage at verification time

Supported circuits (runtime-configured by version):

• transfer - Private transfers (2 inputs → 2 outputs)
• unshield - Withdraw from shielded pool
• value_proof - Note value binding proof (used by claim_shielded_fees)
• private_link - Private identity link proof verification

Runtime vs artifacts

The node verifies proofs against VKs from on-chain storage. Client-side tooling (wallets, dApps) may still require circuit artifacts for proof generation.

RPC Configuration for Privacy Operations

Orbinum extends standard Substrate RPC with privacy-specific methods for querying the shielded pool.

Enable shielded pool RPC:

--rpc-methods Safe  # Already includes shieldedPool_* methods

Available privacy RPCs:

• shieldedPool_getMerkleTreeInfo() - Get current Merkle tree state
• shieldedPool_getMerkleProof(leaf_index) - Get proof for specific commitment
• shieldedPool_getMerkleProofForCommitment(commitment) - Find proof by commitment

EVM RPC (optional for dApp integration):

--ethapi=debug,trace,txpool  # Enable Ethereum debug APIs

Data Directory Structure

Orbinum follows standard Substrate conventions. Inside the Docker container the base path is /data, persisted to a named volume (validator-data / rpc-node-data):

/data/
├── chains/
│   └── orbinum_testnet/           # chain id from testnet-spec.json
│       ├── db/                    # RocksDB (state + blocks)
│       │   ├── full/              # State database
│       │   └── parachains/        # (unused - Orbinum is not a parachain)
│       ├── keystore/              # Session keys (validators only)
│       └── network/               # P2P node identity and DHT data

Testnet Configuration

For testnet setup, see docker/testnet/ in the repository: docker-compose.yml (validator) and docker-compose.rpc.yml (RPC / bootnode), plus the chain spec testnet-spec.json.

---

Validator Registration

Validator onboarding in Orbinum uses a two-phase model managed by pallet-validator-set: candidates self-register by locking a bond, and governance (sudo or council) explicitly approves each entry into the active set.

Minimum ORB Bond

To call registerValidator, the candidate account must hold more than 1,001 ORB in free balance:

Allocation	Amount
Bond locked on-chain	1,000 ORB
Fee buffer (minimum)	1 ORB

The bond is reserved at registration time and returned in full upon voluntary deregistration, governance rejection, or forced removal. Validators added directly via sudo (add_validator) bypass the bond.

Bond storage

The exact bond amount reserved at registration is stored per-account. If the ValidatorBond constant is changed later by governance, the stored value still governs your specific release.

Prerequisites

registerValidator enforces two on-chain prerequisites. The call is rejected if either check fails:

Prerequisite	Check	How to satisfy
Session keys	has_session_keys — Aura + GRANDPA keys registered via session.setKeys	Insert keys into keystore, then call session.setKeys
EVM relay address	has_relayer — an EVM address registered via relayer.registerRelayer	Sudo/governance registers it on your behalf (see below)

EVM Key Auto-Registration

Validators act as EVM relayers by default. This means private transaction users never need to sign their own relay transactions — the active validator node relays them. The EVM address is derived automatically from the Aura session key.

After inserting the Aura key into the keystore and restarting the node, the node reads the key at block #1 and prints:

╔══════════════════════════════════════════════════╗
║   EVM relay key detected — register via sudo     ║
╠══════════════════════════════════════════════════╣
║  Substrate : 5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNe...
║  EVM addr  : 0xd43593c715fdd31c61141abd04a99fd...
╠══════════════════════════════════════════════════╣
║  relayer.registerRelayer(who, evmAddress)        ║
╚══════════════════════════════════════════════════╝

The Orbinum team (or governance) then calls relayer.registerRelayer(who, evmAddress) on-chain using the values from the log. The candidate does not call any extrinsic for this step.

Full Registration Flow (Testnet)

1.  Generate Aura (Sr25519) and GRANDPA (Ed25519) session keys from a single mnemonic
2.  Start the node and wait for sync
3.  Insert keys into keystore via local RPC (author_insertKey — not an on-chain tx)
4.  Restart the node → EVM address is derived and logged automatically
5.  Share the logged Substrate address and EVM address with the Orbinum team
6.  Team registers EVM address via sudo: relayer.registerRelayer(who, evmAddress)
7.  Call session.setKeys on-chain with the combined key hex
8.  Call validatorSet.registerValidator() — locks 1,000 ORB bond, enters pending queue
9.  Governance approves: validatorSet.approveValidator(who)
10. Active at the next session rotation (~1 h on testnet)

Order matters

Steps 7 and 8 must be completed in this exact order. registerValidator checks for both session keys and EVM relay registration. Missing either will result in an on-chain error.

Docker Setup (Testnet — Recommended)

The testnet ships as a pre-built image — you pull it, you do not build it. The fastest way to run a testnet validator:

# Authenticate with GHCR once (PAT with read:packages scope)
docker login ghcr.io -u <your_github_username>

# Clone the repo and configure .env (set VALIDATOR_NAME and VALIDATOR_NODE_KEY)
git clone https://github.com/orbinum/node.git
cd node/docker/testnet
cp .env.example .env   # then edit: VALIDATOR_NODE_KEY=$(openssl rand -hex 32)

# Pull the pre-built image and start (validator + watchtower)
docker compose pull
docker compose up -d
docker compose logs -f orbinum-validator

Wait for Idle in the logs before inserting keys. The docker-compose.yml already sets --validator, --no-mdns, --reserved-only, and the chain spec.

Generate session keys using the helper script (scripts/validator-keys/):

cd scripts/validator-keys
./generate-validator-keys.sh
# Output saved to keys/validator-1/: aura.json, grandpa.json, summary.txt

Insert keys into the running node. The validator's RPC port 9944 is not published to the host — it only listens inside the container. Insert keys with docker exec orbinum-validator curl ... http://localhost:9944, either directly or via the helper:

cd scripts/validator-keys
./insert-session-keys.sh

Each call returns {"result":null} on success. After inserting, restart the node to trigger EVM relay address derivation:

docker compose restart orbinum-validator
docker compose logs -f orbinum-validator   # look for the EVM relay address log box

Full walkthrough

The complete validator onboarding (server setup, GHCR auth, key insertion, on-chain registration) lives in the node repo's docker/testnet/README.md.

Governance-Only Paths

The following extrinsics are restricted to AddRemoveOrigin (sudo / council):

Extrinsic	Effect
add_validator(who)	Directly adds a trusted validator — no bond required
approve_validator(who)	Approves a pending self-registration; takes effect next session
reject_validator(who)	Rejects pending registration; bond returned immediately
remove_validator(who)	Force-removes from pending or active set; bond returned if held

---

Configuration Reference

The testnet flags are set for you in the compose files. The reference below documents what each node type actually runs.

Validator node (docker-compose.yml)

--chain /chain-specs/testnet-spec.json
--name "${VALIDATOR_NAME}"
--base-path /data
--listen-addr /ip4/0.0.0.0/tcp/30333
--validator
--node-key ${VALIDATOR_NODE_KEY}
--reserved-only            # accept P2P only from --reserved-nodes peers
--rpc-port 9944            # listens inside the container only — never published
--rpc-methods Unsafe       # only reachable via `docker exec`, used for key insertion
--no-mdns
--prometheus-external
--prometheus-port 9615

The validator does not expose RPC to the host or the internet. Port 9944 listens only inside the container and is reached with docker exec orbinum-validator curl ... http://localhost:9944 during key insertion. See Security Hardening for the reserved-only + private-network model.

RPC / bootnode (docker-compose.rpc.yml)

--chain /chain-specs/testnet-spec.json
--name "${RPC_NAME}"
--base-path /data
--listen-addr /ip4/0.0.0.0/tcp/30333
--node-key ${RPC_NODE_KEY}     # must derive the PeerId baked into bootNodes
--rpc-port 9944
--rpc-cors all
--rpc-external
--rpc-methods Safe             # read-only — recommended for public endpoints
--rpc-max-connections 1000
--state-pruning archive        # full historical state (explorers/indexers)
--blocks-pruning archive
--no-mdns
--no-private-ipv4
--prometheus-external
--prometheus-port 9615

RPC nodes use host networking and sit behind Caddy + Cloudflare (see RPC Endpoint Protection).

RPC method safety

Public RPC nodes run --rpc-methods Safe. Never expose --rpc-methods Unsafe (which enables author_*) through Cloudflare. If a backend needs unsafe methods (e.g. a faucet), run a separate node reachable only over the private network / loopback.

---

Monitoring and Observability

Prometheus Metrics

Orbinum exposes Substrate and custom metrics on the --prometheus-port (default: 9615).

Key metrics to monitor:

Metric	Description	Alert Threshold
substrate_block_height{status="best"}	Latest block received	-
substrate_block_height{status="finalized"}	Latest finalized block	Lag > 100 blocks
substrate_sub_libp2p_peers_count	Connected peers	< 5 peers
substrate_sub_txpool_validations_finished	Processed transactions	-
process_cpu_seconds_total	CPU usage	> 80% sustained
process_resident_memory_bytes	RAM usage	> 90% of available

Prometheus scrape configuration:

scrape_configs:
  - job_name: 'orbinum-node'
    scrape_interval: 15s
    static_configs:
      - targets: ['localhost:9615']
        labels:
          instance: 'testnet-rpc-1'

Health Check Endpoints

System health:

curl -H "Content-Type: application/json" \
  -d '{"id":1, "jsonrpc":"2.0", "method": "system_health"}' \
  http://localhost:9944

Expected response:

{
  "jsonrpc": "2.0",
  "result": {
    "isSyncing": false,
    "peers": 23,
    "shouldHavePeers": true
  },
  "id": 1
}

Sync status:

curl -H "Content-Type: application/json" \
  -d '{"id":1, "jsonrpc":"2.0", "method": "system_syncState"}' \
  http://localhost:9944

Alerting Rules

Example Prometheus alerts (/etc/prometheus/alerts/orbinum.yml):

groups:
  - name: orbinum-critical
    interval: 30s
    rules:
      - alert: NodeOffline
        expr: up{job="orbinum-node"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Orbinum node {{ $labels.instance }} is offline"

      - alert: LowPeerCount
        expr: substrate_sub_libp2p_peers_count < 5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Node has only {{ $value }} peers"

      - alert: FinalityStalled
        expr: |
          (substrate_block_height{status="best"} - 
           substrate_block_height{status="finalized"}) > 100
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Finality lagging {{ $value }} blocks behind"

      - alert: HighMemoryUsage
        expr: |
          (process_resident_memory_bytes / 
           node_memory_MemTotal_bytes) > 0.9
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Memory usage at {{ $value | humanizePercentage }}"

---

Maintenance Operations

Database Management

Pruning

The public RPC / bootnode runs in archive mode (--state-pruning archive --blocks-pruning archive) so explorers and indexers can query full history. Validators use the chain spec defaults.

Check database size (inside the container):

docker exec orbinum-rpc-node du -sh /data/chains/orbinum_testnet/db/

Backups

Chain data lives in named Docker volumes (validator-data, rpc-node-data). The keystore is the only thing a validator cannot regenerate — back it up.

Validator keystore backup (CRITICAL):

# Copy the keystore out of the container, then encrypt it
docker cp orbinum-validator:/data/chains/orbinum_testnet/keystore ./keystore-backup
tar -czf keystore-$(date +%Y%m%d).tar.gz keystore-backup
gpg --symmetric --cipher-algo AES256 keystore-*.tar.gz
rm -rf keystore-backup

Validator Key Security

Never expose validator keys over RPC or network. The session mnemonic and keystore are what authorize block production — store backups in encrypted, offline storage. Re-importing the same keys onto a second running node risks equivocation.

Software Updates

The testnet image is updated by publishing a new ghcr.io/orbinum/node:testnet-latest to GHCR. Watchtower (included in both compose stacks, gated by the com.centurylinklabs.watchtower.enable label) polls every 5 minutes and rolling-restarts the node automatically — no manual rebuild.

Force an immediate update without waiting for Watchtower:

cd node/docker/testnet
docker compose pull && docker compose up -d        # validator
# RPC: docker compose -f docker-compose.rpc.yml pull && \
#      docker compose -f docker-compose.rpc.yml up -d

Check update history:

docker compose logs orbinum-watchtower

Runtime upgrades:

• Runtime upgrades occur via on-chain governance
• Nodes automatically apply new runtimes when finalized
• No manual intervention required
• Monitor logs for ✨ Imported #X with runtime upgrade messages

---

Troubleshooting

Common Issues

Node Won't Start

Symptoms: Container exits immediately after docker compose up

Diagnosis:

docker compose logs --tail 50 orbinum-validator

# Common errors:
# - "Address already in use" → P2P port 30333 conflict on the host
# - chain-spec mount errors  → testnet-spec.json not mounted / wrong path

Solutions:

# Free port 30333 (find the conflicting process)
sudo ss -ltnp | grep 30333

# Recreate after editing .env / compose
docker compose up -d --force-recreate

Sync Stalled or No Peers

Symptoms: Block height not increasing, peers: 0 for an extended period

Diagnosis (RPC node — RPC is reachable on loopback):

curl -s http://127.0.0.1:9944 -H "Content-Type: application/json" \
  -d '{"id":1,"jsonrpc":"2.0","method":"system_health"}' | jq '.result.peers'

For a validator, RPC is only inside the container:

docker exec orbinum-validator curl -s -H "Content-Type: application/json" \
  -d '{"id":1,"jsonrpc":"2.0","method":"system_health"}' http://localhost:9944 | jq

Solutions:

# Confirm P2P port 30333 is reachable
nc -zv <PUBLIC_OR_PRIVATE_IP> 30333

# Validators run --reserved-only: RESERVED_NODES in .env must list all peers,
# or the node isolates itself and GRANDPA loses quorum (see TOPOLOGY.md).

# Force database rebuild (LAST RESORT — wipes the chain data volume)
docker compose down -v
docker compose up -d

High Memory Usage

Symptoms: Node OOM-killed under query load

The RPC node is capped via mem_limit / cpus in docker-compose.rpc.yml (defaults RPC_MEM_LIMIT=6g, RPC_CPUS=3) so a query flood can't starve Caddy and the OS.

Diagnosis:

docker stats orbinum-rpc-node --no-stream

Solutions:

# Tune the caps in .env, then recreate. Leave ~2 GB RAM + 1 core for OS + Caddy.
RPC_MEM_LIMIT=12g
RPC_CPUS=6

---

Security Hardening

The testnet uses two different network postures: validators stay private (no public RPC, P2P only on a private subnet), and RPC / bootnodes sit behind Cloudflare.

Validator Network Security

Orbinum's sentry validators run with --reserved-only and only peer over the Hetzner private network (10.0.0.0/24) — see docker/testnet/TOPOLOGY.md in the node repo.

• No public RPC. Port 9944 listens only inside the container; it is never published to the host or the internet. Key insertion uses docker exec.
• --reserved-only. The validator accepts P2P connections only from its --reserved-nodes peers (the RPC bootnodes + the other validators) and rejects everything else.
• Private-subnet firewall. 30333/tcp is allowed only from the private subnet, not the whole internet:

sudo ufw allow from 10.0.0.0/24 to any port 30333 proto tcp
sudo ufw allow 22/tcp
sudo ufw enable

Reserved-only quorum

RESERVED_NODES must list all peers (both RPCs + the other validators). If it is incomplete, --reserved-only isolates the node and GRANDPA loses finality quorum. See TOPOLOGY.md for the full mesh wiring.

RPC Endpoint Protection

The public RPC endpoint is served behind Cloudflare, with Caddy on the origin terminating TLS using a Cloudflare Origin Certificate (origin.pem / origin.key) — not Let's Encrypt/certbot, not nginx. The custom Caddy image bundles the caddy-ratelimit plugin. Full walkthrough: docs/rpc-node-setup.md in the node repo.

Defense-in-depth layers:

Layer	What it does
Cloudflare proxy (orange-cloud DNS)	L3/L4 + L7 DDoS protection in front of the origin
Cloudflare rate limit	100 requests / 10s per IP (primary)
Cloudflare WAF skip rule	Skips managed bot challenges on the RPC host — otherwise non-browser clients (indexers, wallets, server-side dApps) get JS-challenged and fail
Caddy rate limit	100 / 10s keyed on CF-Connecting-IP (backstop for direct-IP hits)
Origin firewall	80/443 allowed only from Cloudflare IP ranges; 9944 / 9615 denied
Docker resource limits	mem_limit / cpus (defaults RPC_MEM_LIMIT=6g, RPC_CPUS=3) so a flood can't starve the host

Origin firewall — allow Cloudflare only, deny RPC/metrics:

sudo ufw allow 22/tcp      # SSH
sudo ufw allow 30333/tcp   # P2P — public (this node is a bootnode)

# 80/443 only from Cloudflare's edge ranges
for ip in $(curl -s https://www.cloudflare.com/ips-v4); do
  sudo ufw allow from "$ip" to any port 443 proto tcp
  sudo ufw allow from "$ip" to any port 80 proto tcp
done

sudo ufw deny 9944         # RPC — Caddy reaches the node over loopback
sudo ufw deny 9615         # Prometheus — not public
sudo ufw enable

Caddyfile (origin TLS + per-IP rate limit):

{$RPC_DOMAIN:rpc-1.testnet.orbinum.io} {
	tls /etc/caddy/origin.pem /etc/caddy/origin.key

	rate_limit {
		zone rpc {
			key {http.request.header.CF-Connecting-IP}
			events 100
			window 10s
		}
	}

	reverse_proxy localhost:9944
}

In Cloudflare, set SSL/TLS → Full (Strict), the RPC DNS records to Proxied, and add the WAF skip + rate-limiting rules described in docs/rpc-node-setup.md.

Validator-Specific Security

Sentry node architecture (the running testnet setup):

• Validators run behind the private network and never expose P2P or RPC to the internet
• The public RPC / bootnodes act as the sentries — they are the only public entry points
• Validators connect to the RPCs via the public bootNodes and to each other via --reserved-nodes
• This reduces validator exposure to DDoS and direct attacks

Keystore backup: see Maintenance → Backups. Never expose validator keys over RPC or network; store backups encrypted and offline.

---

Learn More

• Installation → - Initial setup and dependencies
• Running a Node → - Quick start guide
• Consensus Mechanism → - Understanding Orbinum consensus