Skip to content

docs(validators): clarify telemetry labels #438

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
MuncleUscles wants to merge 1 commit into
base: main
Choose a base branch
from
+29 −4
Draft

docs(validators): clarify telemetry labels #438

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 24 additions & 3 deletions pages/validators/monitoring.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,12 +94,27 @@ To contribute your node's metrics and logs to the centralized GenLayer Foundatio
**Prerequisites**
- Metrics enabled in `config.yaml` (`endpoints.metrics: true` — default in recent versions).
- Ops port 9153 exposed in docker-compose (`ports: - "9153:9153"`).
- Credentials from the Foundation team (ask in #testnet-asimov):
- Credentials from the Foundation team (ask in the current validator support channel):
- `CENTRAL_MONITORING_URL` — Prometheus remote write URL
- `CENTRAL_LOKI_URL` — Loki push URL
- `CENTRAL_MONITORING_USERNAME` / `CENTRAL_MONITORING_PASSWORD` — Metrics (Prometheus) credentials
- `CENTRAL_LOKI_USERNAME` / `CENTRAL_LOKI_PASSWORD` — Logs (Loki) credentials

<Callout type="warning">
Use the `docker-compose.yaml` and `alloy-config.river` files that ship with the validator node release. Only customize `.env` values such as credentials, `NODE_ID`, and `VALIDATOR_NAME`. Hand-editing the Alloy scrape component or labels can make metrics reach Grafana Cloud while still being invisible to dashboards, alerts, and uptime crediting.
</Callout>

### Telemetry label contract

Foundation dashboards and validator health checks expect the stock Alloy labels. For a single node setup, the resulting `genlayer_node_*` metrics should include:

- `job="prometheus.scrape.genlayer_node"` from the stock `prometheus.scrape "genlayer_node"` component.
- `validator_name="<your validator name>"` from `VALIDATOR_NAME` in `.env`.
- `instance="<your node id or address>"` from `NODE_ID` in `.env`.
- `network="<network name>"` emitted by the node itself.

If you run multiple nodes, every object in `SCRAPE_TARGETS_JSON` must include a stable `instance` and `validator_name`. Keep the scrape component name `genlayer_node` so the `job` label remains compatible with Foundation dashboards.

**Steps**

1. **Create or update .env** (next to your docker-compose.yaml):
Expand Down Expand Up @@ -128,7 +143,7 @@ LOG_FILE_PATTERN=/var/log/genlayer/node*.log
METRICS_SCRAPE_INTERVAL=15s
```

2. **Add or verify the Alloy service in docker-compose.yaml** (copy if missing):
2. **Use the release-provided Alloy service in docker-compose.yaml**. The validator tarball includes the expected service and config wiring; avoid rewriting it by hand. If you are recovering an older install, compare against the current release and restore the stock service before starting telemetry:

```yaml
# Grafana Alloy for both logs and metrics forwarding
Expand Down Expand Up @@ -205,7 +220,7 @@ volumes:
alloy_data:
```

3. **Create or update ./alloy-config.river** (use the provided version — it handles logs and metrics forwarding):
3. **Create or update ./alloy-config.river** by copying the release-provided version. It handles logs and metrics forwarding and preserves the labels used by Foundation dashboards:

```river
// Grafana Alloy Configuration for GenLayer Node Telemetry
Expand Down Expand Up @@ -429,6 +444,12 @@ curl http://localhost:9153/metrics

- **Target showing DOWN in Alloy UI**: Verify your node is running and the ops port (9153) is accessible. Check that `NODE_METRICS_ENDPOINT` in `.env` is correct.

- **Metrics arrive but dashboards do not show your validator**: compare your labels against the telemetry label contract above. A common sign of a modified Alloy config is `job="genlayer_node"` or a missing `validator_name` label. Restore the release-provided `alloy-config.river`, keep `prometheus.scrape "genlayer_node"`, set `VALIDATOR_NAME`/`NODE_ID` in `.env`, then recreate Alloy:

```bash copy
docker compose --profile monitoring up -d --force-recreate alloy
```

### Getting Help

Share Alloy logs when asking for assistance:
Expand Down
6 changes: 5 additions & 1 deletion pages/validators/setup-guide.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -839,4 +839,8 @@ For a quick setup using Docker Compose:

3. **Verify your node is pushing metrics** by checking the [GenLayer Foundation public dashboard](https://genlayerfoundation.grafana.net/public-dashboards/66a372d856ea44e78cf9ac21a344f792)

For detailed monitoring setup including Prometheus metrics, Grafana dashboards, and centralized logging with Alloy, see the [Monitoring Guide](/validators/monitoring).
<Callout type="warning">
Keep the `alloy-config.river` and monitoring service from the validator release tarball. Only change `.env` values such as credentials, `NODE_ID`, and `VALIDATOR_NAME`; custom Alloy labels can make data arrive in Grafana Cloud while still hiding the validator from dashboards and uptime checks.
</Callout>

For detailed monitoring setup including the required Alloy labels, Prometheus metrics, Grafana dashboards, and centralized logging, see the [Monitoring Guide](/validators/monitoring).
Loading