diff --git a/pages/validators/_meta.json b/pages/validators/_meta.json index d1fda8ca..e758dcdd 100644 --- a/pages/validators/_meta.json +++ b/pages/validators/_meta.json @@ -4,5 +4,7 @@ "system-requirements": "System Requirements", "genvm-configuration": "GenVM Configuration", "upgrade": "Upgrade Guide", + "state-recovery": "State Recovery & Re-sync", + "rollup-outage": "Rollup / Sequencer Outage", "changelog": "Changelog" } diff --git a/pages/validators/monitoring.mdx b/pages/validators/monitoring.mdx index 1b5df4db..06da823d 100644 --- a/pages/validators/monitoring.mdx +++ b/pages/validators/monitoring.mdx @@ -78,6 +78,10 @@ curl -s http://localhost:9153/metrics | grep genlayer_node_ For production validators, we recommend setting up a complete monitoring stack with Prometheus and Grafana. This enables real-time visibility into your validator's performance and helps identify issues before they impact your validator's operation. + + Alert on the `/health` `zksync-connectivity` check and on sync lag (`gen_syncing` → `blocksBehind`). These two signals are what turn a silent problem into a paged incident — see [Rollup / Sequencer Outage](/validators/rollup-outage) and [State Recovery & Re-sync](/validators/state-recovery) for what to do when they fire. + + ## Logs and Metrics Forwarding You can forward your logs and metrics to external systems for centralized monitoring and alerting by using the service `alloy` provided in the `docker-compose.yaml` from the extracted tarball. diff --git a/pages/validators/rollup-outage.mdx b/pages/validators/rollup-outage.mdx new file mode 100644 index 00000000..7c265cc8 --- /dev/null +++ b/pages/validators/rollup-outage.mdx @@ -0,0 +1,194 @@ +--- +description: "Incident-response runbook for GenLayer validator operators when the underlying rollup or sequencer is unreachable: detect the outage, understand why validators are not falsely slashed, communicate status, verify, and resume." +--- + +import { Callout } from "nextra-theme-docs"; + +# Rollup / Sequencer Outage: Incident Response + +A GenLayer validator reads chain state from and submits its votes and receipts to the underlying rollup — the GenLayer Chain (a ZKSync-based L2). When that rollup or its sequencer becomes unreachable, your node cannot read the head or land transactions, and consensus across the whole network stalls. This is fundamentally different from your own node being sick: **the fault is upstream and shared by every validator**, so the correct response is coordinated, not unilateral. + + + **The one thing to internalize:** when the rollup is down, validators are **not** falsely slashed for it. Slashing and quarantine are decided and enacted **on-chain** by the consensus contracts — the node has no code path that penalizes itself, and when it cannot reach the chain it **skips its duties rather than casting a bad vote**. If the chain can't finalize epochs, no penalty can be enacted at all. Your node's job during an outage is to **wait and retry**. Do not wipe your database or re-stake in a panic. + + +This runbook follows the incident lifecycle: **detect → communicate → wait & verify → resume → post-incident.** + +--- + +## Precedent: the Bradbury outage (2026-07-01) + +On 2026-07-01 the rollup underlying the **testnet-bradbury** network went down around 14:00 UTC. Validators could no longer reach the L2, so they stopped participating in consensus and showed as **not-active** / quarantined; because no epochs could finalize, no one was penalized, and the network waited for the rollup to be restored before participation resumed. The steps below generalize that response into a repeatable procedure. + + + Two distinct things happen and it helps to keep them apart. **Not-active** is a **node-side** state: a node that cannot reach the chain stops being "synced" and voluntarily skips its duties (it will not vote on stale state). **Quarantine** is an **on-chain** consensus state, driven by missed activity windows (idleness) or appeal outcomes — mechanisms that require the chain to be *advancing*. In a total rollup outage the chain is frozen, so the outage window itself does not manufacture fresh quarantines; the exposure comes at the **edges** — as the chain resumes and idleness is assessed — and from **asymmetric** failures where your node is unreachable while the chain keeps moving. + + + + Whether your validator shows as not-active or quarantined, it is a **holding state**, not a punishment — it reflects that the validator was not participating because it *could not*. Re-activation after the rollup returns is coordinated (see [Resume](#4-resume)) and happens through normal priming, not something you force by re-staking. + + +--- + +## 1. Detect + +The signature of a rollup outage is that **every path to the L2 fails at once** — RPC and WebSocket both — while your node process itself is otherwise healthy (GenVM up, disk fine, LLM reachable). + +### Run doctor + +`doctor` connects to the configured GenLayer Chain RPC and WebSocket URLs with a short timeout and reports each explicitly: + +```bash copy +./bin/genlayernode doctor +``` + +During an outage you will see failures like: + +``` + ✗ GenLayer Chain RPC: Failed to connect to + Error: + Fix: Ensure GenLayer Chain is running and accessible at the configured URL + ✗ GenLayer Chain WebSocket: Failed to connect to + ... + Some GenLayer Chain checks failed! +``` + +The tell that this is an **outage** and not a **misconfiguration**: the URLs are the same ones that worked before, and the error is a connection/dial failure (connection refused, timeout) rather than "Not configured". + +### Check the health endpoint + +The node's `/health` endpoint (default ops port 9153) includes a named `zksync-connectivity` check that runs the same connection probe the keep-alive loop uses. When the rollup is unreachable, that check fails: + +```bash copy +curl -s http://localhost:9153/health +``` + +Wire this into your alerting (see [Monitoring & Telemetry](/validators/monitoring)) so a rollup outage pages you rather than being discovered by hand. + +### Read the logs + +The node's rollup transport classifies connection/timeout failures as **transient** and retries them with backoff rather than crashing. During an outage the recurring log lines are about the connection, not consensus: + +- `Error checking zksync connection` and `Failed to reconnect zksync.` from the WebSocket keep-alive loop, which probes on a ~1-minute cadence. +- `redialed ws rpc client` / `ws rpc redial failed` as the WebSocket RPC client re-dials a stale connection. +- `retrying RPC request due to timeout` (with `attempt` / `max_attempts`) from the HTTP RPC retry path. +- `[TRANSACT] Gas bump retry hit transient RPC error, retaining budget` when a pending submission cannot land — the node backs off and **keeps** its replacement budget rather than burning it. + + + Timeouts, connection failures, and provider degradation are treated as **transient**: they are retried, they do **not** burn the transaction replacement budget, and they do **not** create nonce gaps. Blockchain reverts and client-side mistakes are classified separately and never trip the breaker — so a rollup outage is handled as "wait for the endpoint to come back," exactly as intended. + + + + The transport-layer **circuit breaker** is observable through **metrics, not logs**. Watch `genlayer_rollup_rpc_circuit_state` (0 = closed, 1 = half-open, 2 = open) and `genlayer_rollup_rpc_circuit_transitions_total` on your metrics endpoint. When gated, calls return a `transport circuit breaker is open` transient error internally. Its thresholds are currently compiled-in constants and cannot be retuned from `config.yaml`. + + +### How the node protects itself while the L2 is down + +You do not need to intervene to prevent damage — the transport layer already backs off intelligently: + +| Mechanism | Behavior during an outage | +|---|---| +| **Retry classification** | Connection/timeout errors are marked transient and retried with backoff; they do not exhaust the replacement budget or leave nonce gaps. | +| **Adaptive throughput** | As the failure rate climbs, request concurrency is cut and delays grow (near-full speed → moderate backoff → one-at-a-time with long waits), so the node stops hammering a dead or degraded endpoint. | +| **Circuit breaker** | After consecutive provider failures the breaker **opens** and gates calls, moves to **half-open** to probe, and **closes** again on success. A write-only fault gates only submissions but keeps reads flowing; a total fault gates everything because the endpoint is presumed dead. | +| **Duty gating** | When the node can no longer track the chain head it stops being "synced," **skips its consensus duties, and pauses its sync pipeline** — it will not vote on stale state. This is the node going *not-active*; it is protective, not a fault. | +| **Keep-alive / reconnect** | The WebSocket connection and RPC client are monitored and automatically re-established when the endpoint returns. | + + + **Do not treat a rollup outage as a local-node problem.** Deleting your chain database, re-running the wizard, or re-staking will not help — the L2 is down for everyone — and a needless database wipe forces a slow [full re-sync](/validators/state-recovery#path-b--full-re-sync-from-genesis-trustless) once the network returns. Confirm the outage is upstream first. + + +--- + +## 2. Communicate + +A rollup outage is a network event. Coordinated communication prevents every operator from independently debugging the same shared failure. + +- **Confirm scope.** Check whether other validators are seeing the same connection failures. If your node's process is healthy (`/health` fails **only** on `zksync-connectivity`) and the RPC/WS URLs are unchanged, the outage is upstream. +- **Report in the operator channel.** Post in the network's operator channel (for testnet, `#testnet-asimov` / `#testnet-bradbury`) with: the UTC time you first saw failures, your node version, and the exact `doctor` / log error strings. This helps the Foundation confirm and triage. +- **Watch for a status update.** The Foundation team coordinates the rollup restart. Expect an acknowledgement of the outage, and later a resume signal. Do not act on rumors — wait for the coordinated all-clear. + + + If your node is forwarding telemetry to the Foundation Grafana Cloud (see [Centralized Push](/validators/monitoring#centralized-push-to-genlayer-foundation-grafana-cloud-using-alloy)), your `zksync-connectivity` failures are already visible in aggregate — another reason to enable it. + + +--- + +## 3. Wait & Verify + +While the rollup is down there is no useful consensus work to do. The goal of this phase is to keep the node **ready** so it rejoins cleanly the moment the L2 returns. + +- **Leave the node running.** Its transport keeps retrying and reconnecting automatically. Stopping it gains nothing and risks a cold start later. +- **Do not modify the database, config, or stake.** Nothing about your node caused the outage; changing things now only adds a second variable to debug. +- **Keep watching detection signals.** `curl http://localhost:9153/health` and `./bin/genlayernode doctor` are your readiness probes — they flip back to healthy the moment the rollup is reachable again. +- **Confirm the outage is genuinely upstream, not you.** If *other* validators recover and you do not, re-check your own RPC/WS URLs, firewall, and DNS — at that point it may be a local connectivity problem masquerading as an outage. See [Node State Recovery & Re-sync](/validators/state-recovery) if your node fell behind but the network is up. + +--- + +## 4. Resume + +Recovery is designed to be **hands-off on the node side**. + +### The node reconnects on its own + +Once the rollup RPC and WebSocket are reachable again: + +1. The keep-alive loop re-establishes the WebSocket subscription. +2. The transport circuit breaker probes (**half-open**) and, on a successful call, returns to **closed** — normal request flow resumes. +3. The node catches up any blocks it missed during the outage. Track progress with `gen_syncing`: + +```bash copy +curl -s -X POST http://localhost:9151 \ + -H "Content-Type: application/json" \ + -d '{"jsonrpc":"2.0","method":"gen_syncing","params":[],"id":1}' +``` + +Wait for `blocksBehind` to return to `0`. If the outage was long and your node fell far behind, follow [Node State Recovery & Re-sync](/validators/state-recovery) to bring it back to the tip. + +### Re-entering the active set + +Validators that were moved to **quarantined / not-active** during the outage rejoin through the network's normal activation path — **priming**, not re-staking. Your stake and identity are untouched. + +```bash copy +# Where does my validator stand after the outage? +genlayer staking active-validators +genlayer staking quarantined-validators +genlayer staking validator-info --validator 0xYourValidatorWallet... +``` + +- Validators must be **primed** each epoch (`validatorPrime()`) to be eligible for selection. Your node **retries priming on its own** until it succeeds — while the chain is down you will see log lines like `Validator could not prime for current epoch. It will not validate until primed. Retrying prime.`, and once the chain is back and the previous epoch has finalized, priming lands and duties resume automatically. See [Validator Priming](/understand-genlayer-protocol/core-concepts/optimistic-democracy/staking#validator-priming). +- Quarantine is managed **on-chain by the consensus contracts**, not by a node-side flag you can clear. A temporary idleness quarantine has an on-chain end epoch and lifts by itself once that epoch passes; lifting one early is an administrative / tribunal action, **not** an operator self-service command. Follow the Foundation's coordinated resume — do not try to force re-activation unilaterally. + + + Because re-activation is coordinated across the whole validator set, resume the way the Foundation directs. Independently re-staking or wiping state does not speed up your return and can leave you worse off (a needless full re-sync). + + +--- + +## 5. Post-Incident + +- **Verify participation.** Confirm the node is producing the expected consensus activity in [monitoring](/validators/monitoring) and that `gen_syncing` reports `blocksBehind: 0`. +- **Confirm you are in the active set.** `genlayer staking active-validators` should list your validator again; it should no longer appear in `quarantined-validators`. +- **Improve detection.** If the outage surprised you, add an alert on the `/health` `zksync-connectivity` check and on `gen_syncing` `blocksBehind`, so the next rollup outage pages you within seconds. +- **Keep a timeline.** Record when you detected the outage, what the errors were, and when you recovered — it shortens triage next time and helps the Foundation correlate reports. + +--- + +## Quick Reference + +| Situation | Action | +|---|---| +| `doctor` shows RPC **and** WebSocket connection failures, URLs unchanged | Rollup outage — proceed with this runbook | +| `doctor` shows "Not configured" | Configuration problem, not an outage — fix the URL | +| Only your node fails; others are fine | Local connectivity issue — check your firewall/DNS/RPC, then [State Recovery](/validators/state-recovery) | +| Rollup back, `blocksBehind` shrinking | Normal — the node is catching up; wait | +| Rollup back, `blocksBehind` stuck/large | Node fell too far behind — see [State Recovery](/validators/state-recovery) | +| Validator shows as not-active / quarantined after outage | Expected — rejoin via automatic priming on the coordinated resume, **not** re-staking | + +## Related + +- [Monitoring & Telemetry](/validators/monitoring) — alert on `/health` and sync lag so outages page you +- [Node State Recovery & Re-sync](/validators/state-recovery) — bring a node that fell behind back to the tip +- [Setup Guide](/validators/setup-guide) — network RPC/WebSocket and consensus configuration +- [Slashing](/understand-genlayer-protocol/core-concepts/optimistic-democracy/slashing) — what slashing actually penalizes (and why an outage does not) +- [Staking](/understand-genlayer-protocol/core-concepts/optimistic-democracy/staking) — priming, the active set, and quarantine diff --git a/pages/validators/state-recovery.mdx b/pages/validators/state-recovery.mdx new file mode 100644 index 00000000..27f547e9 --- /dev/null +++ b/pages/validators/state-recovery.mdx @@ -0,0 +1,256 @@ +--- +description: "Recover a GenLayer validator that has fallen behind or lost state: seed a new node from a snapshot, or trustlessly rebuild the chain database from genesis by re-executing history through GenVM." +--- + +import { Callout } from "nextra-theme-docs"; + +# Node State Recovery & Re-sync + +This runbook covers a validator whose local chain database is behind, corrupted, or gone — a wiped disk, a restore onto new hardware, a database that fell too far behind the chain tip, or a node stuck after an interrupted upgrade. There are two ways to bring a node back to a usable state: + +| Path | What it does | When to use it | Trust model | +|---|---|---|---| +| **Snapshot seeding** (fast) | Populate a new node's data directory from a snapshot captured on a node you already run | You operate more than one node, or you took a snapshot before the incident | Trusts the source node's state | +| **Full re-sync from genesis** (trustless) | Start with an empty database; the node re-scans the rollup and re-executes all history through GenVM | You have no snapshot, or you want an independently verified state | Trustless — re-derives state from the chain | + + + Both paths preserve your on-chain identity. Your validator wallet, stake, and operator key live on the rollup and in your keystore — they are **not** part of the chain database. Recovery only rebuilds local state; it never re-stakes. See [Backing Up Your Operator Key](/validators/setup-guide#backing-up-your-operator-key). + + + + **Tooling is thin on the restore side.** The node exposes a first-class endpoint to **create** a snapshot, but there is no `restore` / `import` command or endpoint that consumes one. Seeding from a snapshot is a manual filesystem operation performed while the node is stopped, described below. If you have no snapshot, the supported recovery is a full re-sync from genesis. + + +--- + +## First: Diagnose How Far Behind You Are + +Before choosing a path, confirm the symptom. A node that is merely catching up will recover on its own — you may not need to do anything. + +### Check GenVM sync status + +The `gen_syncing` RPC method reports how far the node's local state is behind the chain tip: + +```bash copy +curl -s -X POST http://localhost:9151 \ + -H "Content-Type: application/json" \ + -d '{"jsonrpc":"2.0","method":"gen_syncing","params":[],"id":1}' +``` + +```json +{ + "jsonrpc": "2.0", + "result": { + "syncedBlock": "0x151e00", + "latestBlock": "0x151ec5", + "blocksBehind": 197 + }, + "id": 1 +} +``` + +`blocksBehind == 0` means fully synced. A large, **steadily shrinking** `blocksBehind` means the node is catching up normally — let it run. A value that is stuck or growing means the node cannot make progress and needs intervention. + +### Check health and run diagnostics + +```bash copy +# Ops health endpoint (default ops port 9153) +curl -s http://localhost:9153/health + +# Full configuration and connectivity check +./bin/genlayernode doctor +``` + +`doctor` validates consensus contract configuration, GenVM module health, LLM provider connectivity, and rollup (ZKSync) node accessibility — a stuck sync is often an upstream problem (unreachable rollup RPC, dead GenVM modules) rather than a corrupt database. If the rollup itself is unreachable, this is a different incident — see [Rollup / Sequencer Outage](/validators/rollup-outage). + + + A node that falls far enough behind (thousands of blocks, or no sync progress for several minutes) enters a **hopelessly-behind** state: it skips consensus duties and pauses its sync pipeline rather than voting on stale state. This is expected protective behavior. It leaves that state automatically once it makes forward progress, and an internal stall-watchdog restarts the sync pipeline if it wedges — but a node that can *never* catch up (for example, replay is slower than the chain advances, or a dependency is down) is the case this runbook addresses. If a node stays stuck with the pipeline paused, a **process restart** is the historical remedy of last resort. + + +--- + +## Path A — Seed a New Node From a Snapshot (Fast) + +Use this when you run more than one node, or you captured a snapshot before the incident. It restores a node in minutes instead of the hours a full replay can take, at the cost of trusting the source node's state. + +### Step 1 — Create a snapshot on a healthy node + +On a node that is fully synced, call the operations-port `POST /snapshot` endpoint. This produces a consistent snapshot of the chain database at the current block. Full details, authentication, and all modes are documented in the node repository's [snapshot API reference](https://github.com/genlayerlabs/genlayer-node/blob/main/docs/api/ops/snapshot.md). + +The endpoint requires the operator token (`GENLAYERNODE_NODE_OPERATOR_TOKEN`) and is intended for in-host or in-cluster access. From the healthy node, stream the snapshot to a file: + +```bash copy +curl -X POST http://127.0.0.1:9153/snapshot \ + -H "X-Admin-Auth: $GENLAYERNODE_NODE_OPERATOR_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{}' \ + -o snapshot.tar.gz +``` + + + The endpoint returns `401` to every caller when `GENLAYERNODE_NODE_OPERATOR_TOKEN` is unset on the node (default-deny). If you have never set this variable, set it and restart the node before you can take a snapshot. Treat the token and the resulting archive as sensitive — anyone holding the archive can reconstruct the node's state. + + +The snapshot is a gzipped tar of the chain database at the moment of capture: the SSTables, `MANIFEST`, and write-ahead log (WAL). It can be several gigabytes, so the connection stays open for the duration — configure any reverse proxy with a generous read timeout and disable response buffering. + + + If your snapshot-taking node and target node share a filesystem (for example a mounted volume), the endpoint can also write the snapshot directly to a path inside the configured snapshot directory (`node.admin.snapshot_dir`, default `/snapshots`) instead of streaming it. See the [snapshot API reference](https://github.com/genlayerlabs/genlayer-node/blob/main/docs/api/ops/snapshot.md) for write-to-path mode. + + +### Step 2 — Stop the target node + +Seeding is an offline operation. The database must not be written while you replace it. + +```bash copy +# If running via docker-compose +docker compose --profile node down + +# If running the binary directly, stop the genlayernode process +``` + +### Step 3 — Place the snapshot into the data directory + +The snapshot archive is a consistent checkpoint of the node's Pebble chain database (`node.genlayerdb`, default `data/node/genlayer.db`) — the SSTables, `MANIFEST`, and WAL. Extract it back into place under the target node's `datadir`. + + + **Inspect before you extract.** Because there is no `restore` command that validates the archive, extract to a scratch directory first and confirm the layout matches what your `config.yaml` expects before copying it into `datadir`. Never overwrite `data/node/keystore/` — that is your own operator key. + + +```bash copy +# 1. Inspect the archive so you know its internal layout +tar -tzvf snapshot.tar.gz | head + +# 2. Back up the current (behind/corrupt) chain DB so you can roll back +mv ./data/node/genlayer.db ./data/node/genlayer.db.bak 2>/dev/null || true + +# 3. Extract the checkpoint into the datadir, matching node.genlayerdb +tar -xzvf snapshot.tar.gz -C ./data/node +``` + + + The archive checkpoints the **Pebble chain DB**. The `merkleforest` / `merkletree` stores are separate paths under `datadir`; if your snapshot workflow does not carry them, the node rebuilds any missing derived state on start and then syncs the remaining gap. Confirm the extracted directory name matches `node.genlayerdb` in your `config.yaml`, and that the `datadir` you extracted into is the one the node runs against. + + +### Step 4 — Start the node and confirm it converges + +```bash copy +./bin/genlayernode doctor +./bin/genlayernode run -c $(pwd)/configs/node/config.yaml --password "your secret password" +``` + +The node starts from the snapshot's block and syncs the gap to the current tip. Poll `gen_syncing` until `blocksBehind` reaches 0: + +```bash copy +watch -n 5 'curl -s -X POST http://localhost:9151 \ + -H "Content-Type: application/json" \ + -d "{\"jsonrpc\":\"2.0\",\"method\":\"gen_syncing\",\"params\":[],\"id\":1}"' +``` + +Once caught up, verify the validator is participating in [monitoring](/validators/monitoring) and, if it had been excluded from selection while down, that it is being primed again (see [Re-entering the Validator Set](#re-entering-the-validator-set-after-downtime)). + +--- + +## Path B — Full Re-sync From Genesis (Trustless) + +Use this when you have no snapshot, or when you want a state you derived yourself rather than one you accepted from another node. The node starts with an empty database, **scans the rollup for the accepted-transaction history, and re-executes every transaction through GenVM**, rebuilding local state block by block. History comes from the GenLayer Chain rollup itself (the node filters the consensus contract's logs) — not from a separate data-availability layer or from peers. This is the same process a brand-new validator goes through on first start, and the same one a breaking upgrade forces (see [Breaking Change Upgrade](/validators/upgrade#breaking-change-upgrade)). + +Because every result is recomputed locally rather than trusted, this path is trustless — but it is bounded by how fast your node can replay history, which is dominated by GenVM execution and your LLM provider's throughput. + +### Step 1 — Stop the node and clear the chain database + +```bash copy +# Stop the node first (docker compose --profile node down, or stop the binary) + +# Remove the chain database so the node rebuilds it from genesis. +# The chain state is the Pebble DB (node.genlayerdb, default data/node/genlayer.db) +# plus the Merkle forest/tree stores under data/node/merkle/. +# Keep configs/ and data/node/keystore/ — only the chain state goes. +rm -rf ./data/node/genlayer.db ./data/node/merkle +``` + + + These are the example-config defaults. Confirm against your own `config.yaml`: the chain database is `node.genlayerdb` under `datadir`, and the Merkle stores are `merkleforest.dbpath`, `merkleforest.indexdbpath`, and `merkletree.dbpath`. Remove exactly those, and nothing else under `datadir`. + + + + **Never delete `data/node/keystore/`.** That is your operator key. If you lose it you must set up a new operator and update your validator configuration. Back it up first — see [Backing Up Your Operator Key](/validators/setup-guide#backing-up-your-operator-key). + + +### Step 2 — Set the genesis block to bound the scan + +Confirm your `config.yaml` sets `consensus.genesis` for your network. This tells the node the first block of the consensus deployment so it does not scan the settlement chain from block 0. + +```yaml copy +consensus: + # Testnet Asimov + consensusaddress: "0xe66B434bc83805f380509642429eC8e43AE9874a" + genesis: 17326 +``` + + + If `genesis` is omitted the node auto-detects it by searching for the first log from the consensus contract, which is slower. Setting it explicitly is the single biggest win for re-sync time. Use the value for **your** network — see [Network-Specific Consensus Configuration](/validators/setup-guide#network-specific-consensus-configuration). + + +### Step 3 — (Optional) Raise replay throughput + +During re-sync the node is doing bulk GenVM replay, which is more parallel than steady-state operation. The `genvm.sync_permits` setting controls how many GenVM permits are used while the node is not yet synced; it defaults to the steady-state `permits` value if left empty. + +```yaml copy +genvm: + permits: 8 # steady-state concurrency + sync_permits: 64 # concurrency during bulk replay (while not synced) + sync_wait_timeout: "20s" +``` + +Raising `sync_permits` speeds up replay only as far as your LLM provider's rate limits and your CPU allow. See [GenVM Configuration](/validators/genvm-configuration) for provider throughput and the greybox fallback. + +### Step 4 — Start the node and monitor replay + +```bash copy +./bin/genlayernode doctor +./bin/genlayernode run -c $(pwd)/configs/node/config.yaml --password "your secret password" +``` + +Watch `gen_syncing` and your logs. `blocksBehind` should fall steadily. Because replay re-runs Intelligent Contracts through your LLM provider, expect this to take significantly longer than a snapshot seed, and to consume LLM credits. + + + If `blocksBehind` is **not** shrinking, the node is not making progress. Re-run `doctor` and check that the rollup RPC is reachable, the GenVM modules are running, and your LLM provider is not rate-limiting you. A stalled replay is almost always an upstream dependency, not the database. + + +--- + +## Re-entering the Validator Set After Downtime + +Recovering the database restores the node — but if your validator was down long enough to miss its per-epoch **priming**, it may have been dropped from validator selection while it was offline. Rebuilding local state does not itself re-select you. + +- Validators must be **primed** each epoch (`validatorPrime()`) to remain eligible for selection. Missing priming does **not** lose your stake or rewards, but it excludes you from selection until priming resumes. See [Validator Priming](/understand-genlayer-protocol/core-concepts/optimistic-democracy/staking#validator-priming). +- A validator that missed activity windows may appear in the **quarantined** set rather than the **active** set. Quarantine is an on-chain consensus mechanism, not a node-side flag — check your status with the CLI: + +```bash copy +# Is my validator active, or quarantined? +genlayer staking active-validators +genlayer staking quarantined-validators +genlayer staking validator-info --validator 0xYourValidatorWallet... +``` + + + If you were offline because the **rollup itself** was down (not just your node), see [Rollup / Sequencer Outage: Incident Response](/validators/rollup-outage) — in that scenario the whole network waits, and re-activation is coordinated rather than something you fix unilaterally. + + +--- + +## Quick Troubleshooting + +- **`gen_syncing` shows `blocksBehind` stuck or growing** — The node cannot keep up or cannot progress. Run `doctor`; check rollup RPC reachability, GenVM module health, and LLM rate limits before assuming database corruption. +- **Node won't start after seeding a snapshot** — The extracted directory layout does not match your `config.yaml` paths. Confirm `merkleforest.dbpath`, `merkletree.dbpath`, and index paths, and that you extracted into the correct `datadir`. +- **`POST /snapshot` returns 401** — `GENLAYERNODE_NODE_OPERATOR_TOKEN` is unset or the header is wrong. Set the token on the node and restart; it is read once at startup and not hot-reloaded. +- **Re-sync is extremely slow** — Expected for a full genesis replay: every historical Intelligent Contract is re-executed through GenVM. Confirm `consensus.genesis` is set, raise `genvm.sync_permits`, and check your LLM provider is not the bottleneck. +- **Recovered node runs but isn't selected for consensus** — You likely missed priming while down. Verify with `genlayer staking active-validators` / `quarantined-validators` and see [Validator Priming](/understand-genlayer-protocol/core-concepts/optimistic-democracy/staking#validator-priming). + +## Related + +- [Setup Guide](/validators/setup-guide) — installation, configuration, and operator-key backup/restore +- [Upgrade Guide](/validators/upgrade) — breaking upgrades also require a fresh sync from genesis +- [Monitoring & Telemetry](/validators/monitoring) — confirm the recovered node is healthy and participating +- [Rollup / Sequencer Outage](/validators/rollup-outage) — when the failure is upstream, not local +- [Staking](/understand-genlayer-protocol/core-concepts/optimistic-democracy/staking) — priming, active set, and quarantine diff --git a/pages/validators/upgrade.mdx b/pages/validators/upgrade.mdx index 0802f772..dc253b9a 100644 --- a/pages/validators/upgrade.mdx +++ b/pages/validators/upgrade.mdx @@ -65,4 +65,4 @@ Breaking changes involve a new consensus deployment. The old database is incompa - **Node won't start** — Run `./bin/genlayernode doctor`. If it reports unknown config fields, reconfigure from the new example. - **Not participating in consensus** — Verify the consensus contract address, `validatorWalletAddress`, and `operatorAddress` are correct. - **Key errors** — Your operator key wasn't imported. See [restoring your operator key](/validators/setup-guide#restoring-your-operator-key). -- **Slow sync after breaking upgrade** — Expected. Ensure `genesis` is set in `config.yaml` to avoid scanning from block 0. +- **Slow sync after breaking upgrade** — Expected. Ensure `genesis` is set in `config.yaml` to avoid scanning from block 0. A breaking upgrade rebuilds the database from genesis exactly like a full re-sync — see [State Recovery & Re-sync](/validators/state-recovery) for how to speed up replay and confirm the node converges.