Skip to content

Add health fault lifecycle tracking and InfluxDB export - #3

Draft
alaurenzi wants to merge 12 commits into
masterfrom
agent/add-fault-lifecycle-tracker
Draft

Add health fault lifecycle tracking and InfluxDB export#3
alaurenzi wants to merge 12 commits into
masterfrom
agent/add-fault-lifecycle-tracker

Conversation

@alaurenzi

@alaurenzi alaurenzi commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

What changed

  • add aggregator-owned fault lifecycle tracking for standardized health reports
  • define a string-native XBot2/ROS diagnostics contract for nodes ending in /health
  • track raises and clears by (hw_id, node, fault_report)
  • preserve hardware fault state when the diagnostics source becomes stale
  • write Grafana-friendly health snapshots and transition-only fault_event points to InfluxDB
  • use standardized fault_report values as InfluxDB tags
  • add focused lifecycle, contract, ordering, timestamp, and Influx schema tests

XBot2 publisher contract

Only diagnostic names whose final path segment is health participate in fault lifecycle tracking.

A valid health status contains exactly one decimal string count and repeated string reports:

name: /xbot/joint/knee_pitch_1/health
hardware_id: knee_pitch_1
level: 2
message: Drive faults active
values:
  - key: fault_count
    value: "2"
  - key: fault_report
    value: motor over temperature
  - key: fault_report
    value: encoder signal lost

The reports are the complete active set, not deltas. fault_count must equal the number of unique reports.

  • no reports + level 0: healthy and clears previous reports
  • reports + level 1: warning reports active
  • reports + level 2: error reports active
  • level 3: source stale; no hardware raise or clear

Malformed or inconsistent messages are ignored for lifecycle purposes rather than clearing existing state.

Publish immediately on set/severity/summary changes and also publish a periodic heartbeat; 1 Hz is the documented default.

InfluxDB schema

health

Written for every health publication, including heartbeats, yielding one latest row per health source for Grafana.

Tags: hw_id, path, name, component.

Fields include level, message, fault_count, active_fault_reports, and—after the first transition—last_fault_report, last_fault_active, last_fault_level, last_raised_ns, and last_cleared_ns when available.

fault_event

Written only on raise/clear transitions.

Tags: hw_id, path, name, component, fault_report, transition.

Fields: active, level, message, occurrence_count, first_raised_ns, last_raised_ns, and last_cleared_ns when available.

fault_report is intentionally a tag because reports are standardized and bounded, with cardinality equivalent to standardized fault codes while remaining directly readable in Grafana.

Both measurements use the source timestamp when valid, with wall-clock fallback.

Documentation and validation

The complete convention and storage schema are documented in docs/fault_health_contract.md.

Tests cover:

  • single and simultaneous reports
  • duplicate heartbeat suppression for events
  • raise, clear, and report replacement
  • stale behavior
  • /health namespace enforcement
  • count and level validation
  • transition-before-snapshot ordering
  • health snapshot fields
  • fault_event report and transition tags
  • source timestamp usage

The branch is currently ten commits ahead of master, with six changed files.

@alaurenzi alaurenzi changed the title Add fault lifecycle tracking to the aggregator Add health fault lifecycle tracking and InfluxDB export Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant