diff --git a/pages/validators/monitoring.mdx b/pages/validators/monitoring.mdx index 1b5df4db..6db255ba 100644 --- a/pages/validators/monitoring.mdx +++ b/pages/validators/monitoring.mdx @@ -94,12 +94,27 @@ To contribute your node's metrics and logs to the centralized GenLayer Foundatio **Prerequisites** - Metrics enabled in `config.yaml` (`endpoints.metrics: true` — default in recent versions). - Ops port 9153 exposed in docker-compose (`ports: - "9153:9153"`). -- Credentials from the Foundation team (ask in #testnet-asimov): +- Credentials from the Foundation team (ask in the current validator support channel): - `CENTRAL_MONITORING_URL` — Prometheus remote write URL - `CENTRAL_LOKI_URL` — Loki push URL - `CENTRAL_MONITORING_USERNAME` / `CENTRAL_MONITORING_PASSWORD` — Metrics (Prometheus) credentials - `CENTRAL_LOKI_USERNAME` / `CENTRAL_LOKI_PASSWORD` — Logs (Loki) credentials + + Use the `docker-compose.yaml` and `alloy-config.river` files that ship with the validator node release. Only customize `.env` values such as credentials, `NODE_ID`, and `VALIDATOR_NAME`. Hand-editing the Alloy scrape component or labels can make metrics reach Grafana Cloud while still being invisible to dashboards, alerts, and uptime crediting. + + +### Telemetry label contract + +Foundation dashboards and validator health checks expect the stock Alloy labels. For a single node setup, the resulting `genlayer_node_*` metrics should include: + +- `job="prometheus.scrape.genlayer_node"` from the stock `prometheus.scrape "genlayer_node"` component. +- `validator_name=""` from `VALIDATOR_NAME` in `.env`. +- `instance=""` from `NODE_ID` in `.env`. +- `network=""` emitted by the node itself. + +If you run multiple nodes, every object in `SCRAPE_TARGETS_JSON` must include a stable `instance` and `validator_name`. Keep the scrape component name `genlayer_node` so the `job` label remains compatible with Foundation dashboards. + **Steps** 1. **Create or update .env** (next to your docker-compose.yaml): @@ -128,7 +143,7 @@ LOG_FILE_PATTERN=/var/log/genlayer/node*.log METRICS_SCRAPE_INTERVAL=15s ``` -2. **Add or verify the Alloy service in docker-compose.yaml** (copy if missing): +2. **Use the release-provided Alloy service in docker-compose.yaml**. The validator tarball includes the expected service and config wiring; avoid rewriting it by hand. If you are recovering an older install, compare against the current release and restore the stock service before starting telemetry: ```yaml # Grafana Alloy for both logs and metrics forwarding @@ -205,7 +220,7 @@ volumes: alloy_data: ``` -3. **Create or update ./alloy-config.river** (use the provided version — it handles logs and metrics forwarding): +3. **Create or update ./alloy-config.river** by copying the release-provided version. It handles logs and metrics forwarding and preserves the labels used by Foundation dashboards: ```river // Grafana Alloy Configuration for GenLayer Node Telemetry @@ -429,6 +444,12 @@ curl http://localhost:9153/metrics - **Target showing DOWN in Alloy UI**: Verify your node is running and the ops port (9153) is accessible. Check that `NODE_METRICS_ENDPOINT` in `.env` is correct. +- **Metrics arrive but dashboards do not show your validator**: compare your labels against the telemetry label contract above. A common sign of a modified Alloy config is `job="genlayer_node"` or a missing `validator_name` label. Restore the release-provided `alloy-config.river`, keep `prometheus.scrape "genlayer_node"`, set `VALIDATOR_NAME`/`NODE_ID` in `.env`, then recreate Alloy: + +```bash copy +docker compose --profile monitoring up -d --force-recreate alloy +``` + ### Getting Help Share Alloy logs when asking for assistance: diff --git a/pages/validators/setup-guide.mdx b/pages/validators/setup-guide.mdx index 08d86e39..ccee2c51 100644 --- a/pages/validators/setup-guide.mdx +++ b/pages/validators/setup-guide.mdx @@ -839,4 +839,8 @@ For a quick setup using Docker Compose: 3. **Verify your node is pushing metrics** by checking the [GenLayer Foundation public dashboard](https://genlayerfoundation.grafana.net/public-dashboards/66a372d856ea44e78cf9ac21a344f792) -For detailed monitoring setup including Prometheus metrics, Grafana dashboards, and centralized logging with Alloy, see the [Monitoring Guide](/validators/monitoring). + + Keep the `alloy-config.river` and monitoring service from the validator release tarball. Only change `.env` values such as credentials, `NODE_ID`, and `VALIDATOR_NAME`; custom Alloy labels can make data arrive in Grafana Cloud while still hiding the validator from dashboards and uptime checks. + + +For detailed monitoring setup including the required Alloy labels, Prometheus metrics, Grafana dashboards, and centralized logging, see the [Monitoring Guide](/validators/monitoring).