meta: expose the background subsystems on /metrics

[bjmeetsfo]

The problem

Conviction, shard-divergence reconciliation, retention, freeze aging and rebalancing all run on background loops and reported what they did only through tracing.

That leaves an operator scraping logs to answer the questions they actually ask during an incident:

• Is the fleet being convicted right now?
• Is any location in safe mode?
• How many shards is the metaserver routing to nodes that do not serve them?
• Is retention keeping up, or is the per-round cap always saturated?

The reference metaserver emits per-tag counters for exactly these (server_convict_count, server_damage_severity, abnormal_server_count, missing_partition_count, balance_partition_count, freeze_partition_count).

What this adds

meta/subsystem_metrics.rs — the recorder those loops write their round outcomes into, rendered onto the metaserver's existing /metrics surface alongside the request and inventory series already there.

It holds two kinds of series, and the distinction is the point:

Counters only climb — resources convicted, convictions held back by a guard, reboots detected, divergences found, shards reassigned by cause, tombstones purged, frozen resources aged out, rounds completed per subsystem. These answer "is this happening, and how often".

Gauges describe the most recent round — per-location damage severity and abnormal count, whether a location is in safe mode, whether a detector is paused, how many shards are diverged or settling, what the retention cap held back. These answer "what is true now", and are replaced wholesale each round: a location that recovers reports zero rather than sticking at its worst value, and a location that disappears stops reporting at all. Both are covered by tests, because a gauge that latches is a gauge that pages after the incident is over.

Design notes

• The recorder lives on SingleNodeMeta and is shared by clone, so every handle writes to the same series and the loops need no extra plumbing.
• Recording happens at each subsystem's entry point, not in the loop bodies — so a direct call, including from a test or an admin route, is counted too.
• The raft backend renders nothing, because it drives none of these loops.
• reassign_shard grows a reason-carrying variant so a move can be attributed. ShardReassignmentReason gains as_str, matching its serde representation so a dashboard and a JSON payload name the same cause the same way. The existing reassign_shard keeps working and records unspecified.
• A poisoned metrics lock recovers rather than propagating: observability is not worth taking the metaserver down for.
• Label values are escaped — locations and reasons are operator-supplied, and a stray quote would otherwise corrupt the exposition format.

Series

Metric	Type	Labels
temporalstore_meta_convicted_total	counter	tier
temporalstore_meta_conviction_held_total	counter	tier, guard
temporalstore_meta_detector_rounds_total	counter	subsystem
temporalstore_meta_reboots_detected_total	counter	—
temporalstore_meta_damage_severity	gauge	tier, location
temporalstore_meta_abnormal_resources	gauge	tier, location
temporalstore_meta_location_safe_mode	gauge	tier, location
temporalstore_meta_detector_paused	gauge	tier
temporalstore_meta_shard_divergence_total	counter	—
temporalstore_meta_shard_divergence	gauge	state
temporalstore_meta_shards_reassigned_total	counter	reason
temporalstore_meta_retention_purged_total	counter	kind
temporalstore_meta_retention_blocked	gauge	—
temporalstore_meta_retention_capped	gauge	—
temporalstore_meta_freeze_aged_total	counter	kind

Cardinality note (also in the module docs): damage is labelled by location, matching the reference's per-tag emission and bounded by the number of distinct locations — but with hierarchical locations (#73) a deployment labelling every rack separately gets one series per rack.

Tests

11 new tests: counters accumulating across rounds, damage gauges describing only the latest round, a location that disappears no longer reporting, the two tiers reporting separately, divergence keeping a running total beside a current state, retention and aging counters climbing by kind, reassignments labelled by cause, label escaping, every series carrying HELP and TYPE, a clone sharing one recorder, and the /metrics route exposing the new series end to end.

Verification:

• cargo test -p temporalstore-rust --lib meta::subsystem_metrics — 10 passed, 0 failed.
• cargo test -p temporalstore-rust --lib meta — 198 passed, 0 failed.
• cargo test -p temporalstore-rust --bin metaserver — 18 passed, 0 failed.
• cargo build -p temporalstore-rust --bin metaserver — clean, no new warnings.

No gate: this only adds series to an endpoint that already exists, and changes no behaviour. Covers the subsystems merged so far; the guards in #75 and #76 can add their own series when those land.