Monitoring & Alerts

Two things worth separating: checking a node right now, and being told when something breaks at 3am. The first is a curl; the second is the monitoring stack.

Health checks

Every role answers the same RPC, whatever its firewall posture. On a node whose RPC is loopback-only, go through the container:

docker exec orbinum-validator curl -s -H 'Content-Type: application/json' \
  -d '{"id":1,"jsonrpc":"2.0","method":"system_health"}' \
  http://localhost:9944

A healthy node returns isSyncing: false and a non-zero peers:

{"jsonrpc":"2.0","result":{"peers":3,"isSyncing":false,"shouldHavePeers":true},"id":1}

Sync progress, when it is still catching up:

docker exec orbinum-validator curl -s -H 'Content-Type: application/json' \
  -d '{"id":1,"jsonrpc":"2.0","method":"system_syncState"}' \
  http://localhost:9944

Compare currentBlock against highestBlock.

Prometheus metrics

Every role exposes metrics on 9615. Where that port listens is the part that differs — see Ports & Endpoints. Keep it on a private interface; the endpoint is unauthenticated, so the firewall is the only thing protecting it.

The metrics worth watching:

Metric	What it tells you
`substrate_block_height{status="best"}`	Chain is still producing
`substrate_block_height{status="finalized"}`	GRANDPA is still finalizing
`substrate_sub_libp2p_peers_count`	Peering health
`substrate_ready_transactions_number`	Transaction pool backlog
`process_resident_memory_bytes`	Memory pressure

The alerting stack

node-deploy ships a complete monitoring stack — use it rather than writing rules by hand. testnet/monitoring/ runs Prometheus, Alertmanager (with Telegram delivery) and Grafana in one Compose file, with alert rules already written and tested.

It is generic: you declare which nodes to watch in prometheus/targets/, so the same stack works for one node on your laptop or a fleet on a private VPC.

Everything binds to 127.0.0.1 on the monitoring host. Reach Grafana and Prometheus over an SSH tunnel:

ssh -L 3000:localhost:3000 -L 9090:localhost:9090 user@<monitoring-host>

Each node host also runs the small node-exporter stack from testnet/monitoring/node-exporter/, which reports CPU, memory and disk.

What it already alerts on

The shipped prometheus/rules/ cover the failures that matter:

Alert	Fires when
`FinalityStalled`	Best and finalized heights diverge
`GrandpaRoundStalled`	No GRANDPA round progress for 5m
`ChainHalted`	No new blocks for 3m
`ChainForkDetected`	More than one leaf for 10m
`ValidatorHeightDivergence`	One validator falls behind the others
`BlockTimeDegraded`	Blocks arriving slower than target
`TransactionPoolBacklog`	Ready transactions above threshold

FinalityStalled is the one to act on first

GRANDPA needs more than two thirds of the set voting. With a small validator set, a couple of nodes offline stops finality for everyone while block production continues — so the chain looks alive from the outside. A growing gap between best and finalized is the signal, and it is the kind of failure that otherwise goes unnoticed for hours.

The monitoring stack deliberately runs no Watchtower: the thing that tells you the chain is broken must not restart itself mid-incident.

Health checks​

Prometheus metrics​

The alerting stack​

What it already alerts on​

Health checks

Prometheus metrics

The alerting stack

What it already alerts on