Add health fault lifecycle tracking and InfluxDB export - #3
Draft
alaurenzi wants to merge 12 commits into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
/health(hw_id, node, fault_report)healthsnapshots and transition-onlyfault_eventpoints to InfluxDBfault_reportvalues as InfluxDB tagsXBot2 publisher contract
Only diagnostic names whose final path segment is
healthparticipate in fault lifecycle tracking.A valid health status contains exactly one decimal string count and repeated string reports:
The reports are the complete active set, not deltas.
fault_countmust equal the number of unique reports.0: healthy and clears previous reports1: warning reports active2: error reports active3: source stale; no hardware raise or clearMalformed or inconsistent messages are ignored for lifecycle purposes rather than clearing existing state.
Publish immediately on set/severity/summary changes and also publish a periodic heartbeat; 1 Hz is the documented default.
InfluxDB schema
healthWritten for every health publication, including heartbeats, yielding one latest row per health source for Grafana.
Tags:
hw_id,path,name,component.Fields include
level,message,fault_count,active_fault_reports, and—after the first transition—last_fault_report,last_fault_active,last_fault_level,last_raised_ns, andlast_cleared_nswhen available.fault_eventWritten only on raise/clear transitions.
Tags:
hw_id,path,name,component,fault_report,transition.Fields:
active,level,message,occurrence_count,first_raised_ns,last_raised_ns, andlast_cleared_nswhen available.fault_reportis intentionally a tag because reports are standardized and bounded, with cardinality equivalent to standardized fault codes while remaining directly readable in Grafana.Both measurements use the source timestamp when valid, with wall-clock fallback.
Documentation and validation
The complete convention and storage schema are documented in
docs/fault_health_contract.md.Tests cover:
/healthnamespace enforcementhealthsnapshot fieldsfault_eventreport and transition tagsThe branch is currently ten commits ahead of
master, with six changed files.