Skip to content

meta: give proxies the same adaptive detection and safe mode as datanodes - #71

Merged
matrixarkai merged 1 commit into
mainfrom
oss/meta-proxy-adaptive-detection
Aug 19, 2026
Merged

meta: give proxies the same adaptive detection and safe mode as datanodes#71
matrixarkai merged 1 commit into
mainfrom
oss/meta-proxy-adaptive-detection

Conversation

@bjmeetsfo

Copy link
Copy Markdown
Collaborator

Closes a hole I left open in #61 on reasoning that turned out to be incomplete.

The problem

#61 deliberately left proxies on the fixed stale_after_ms threshold, arguing that freezing a proxy is cheap because it moves no data.

That covers one of the three problems the detector was built for and misses the two that matter here. A proxy freeze being individually cheap says nothing about the correlated case:

  • The fixed threshold is measured against wall clock, so a metaserver stall makes every proxy cross it at the same instant.
  • A rack fault makes every proxy behind that rack look stale together.

The proxies are the routing tier. Freezing all of them is a total outage of the serving path. The blast radius of a correlated proxy failure is larger than the datanode equivalent, not smaller — which makes proxies the tier that needs the guard most, not the one that can go without it.

The reference metaserver runs phi detection and per-tag safe mode for proxies exactly as it does for servers.

What this does

Proxies now run the same round the datanodes do: phi-accrual detection against each proxy's own learned heartbeat cadence, the stall guard that suppresses conviction after the detector itself pauses, and the per-location safe mode that holds back conviction when too much of one location is failing at once.

The round body is factored into one shared function over a ConvictionSubject, so the two tiers cannot drift apart — a change to detection applies to both by construction.

Each tier gets its own detector instance. Datanode heartbeat cadence says nothing about proxy cadence, so sharing one would judge each against the other's rhythm; separate stall clocks also mean a pause is judged against the tier it actually affected. A test asserts the two do not share distributions.

A proxy carries no boot-time anchor, so the reboot path (#64) never fires for one — only silence convicts a proxy.

Variable Default Meaning
TS_META_CONVICT_PROXIES 1 Convict proxies as well as datanodes

With it off, damage is still assessed and reported, so a failing proxy stays visible either way. It is a separate knob from TS_META_CONVICT_ENABLED because the two tiers fail for different reasons and an operator may want to hold the routing tier steady while datanode conviction runs.

The adaptive loop now drives both tiers and no longer falls back to freeze_stale_proxies; that function stays in place for the default fixed-threshold path, which is unchanged.

Tests

7 new tests: a single silent proxy convicted, a whole-rack proxy failure held back, the stall guard suppressing proxy conviction, conviction switchable off while damage is still reported, a frozen proxy counting as damage without being reconvicted, proxies never being convicted for a restart, and the two tiers not sharing a heartbeat distribution.

Verification:

  • cargo test -p temporalstore-rust --lib meta::failure_detector — 28 passed, 0 failed.
  • cargo test -p temporalstore-rust --lib meta159 passed, 0 failed. (One earlier run showed the two known-flaky proxy::tests failures that also fail on pristine main; they passed on re-run.)
  • cargo test -p temporalstore-rust --bin metaserver — 18 passed, 0 failed.
  • cargo build -p temporalstore-rust --bin metaserver — clean, no new warnings.

Still off by default behind TS_META_ADAPTIVE_FAILURE_DETECTOR.

…odes

The adaptive failure detector deliberately left proxies on the fixed
stale_after_ms threshold, on the grounds that freezing a proxy is cheap because
it moves no data. That reasoning covers one of the three problems the detector
was built for and misses the two that matter here.

A proxy freeze being individually cheap says nothing about the correlated case.
The fixed threshold is measured against wall clock, so a metaserver stall makes
every proxy cross it at the same instant, and a rack fault makes every proxy
behind that rack look stale together. The proxies are the routing tier: freezing
all of them is a total outage of the serving path. The blast radius of a
correlated proxy failure is larger than the datanode equivalent, not smaller,
which makes the proxies the tier that needs the guard most.

Proxies now run the same round the datanodes do: phi-accrual detection against
each proxy's own learned heartbeat cadence, the stall guard that suppresses
conviction after the detector itself pauses, and the per-location safe mode that
holds back conviction when too much of one location is failing at once.

The round body is factored into one shared function over a ConvictionSubject, so
the two tiers cannot drift apart - a change to detection applies to both by
construction. A proxy carries no boot-time anchor, so the reboot path never
fires for one; only silence convicts a proxy.

Each tier gets its own detector instance. Datanode heartbeat cadence says
nothing about proxy cadence, so sharing one would judge each against the other's
rhythm, and separate stall clocks mean a pause is judged against the tier it
actually affected.

ConvictionPolicy::convict_proxies (on by default, TS_META_CONVICT_PROXIES to
disable) lets an operator hold the routing tier steady while datanode conviction
runs. With it off the damage is still assessed and reported, so a failing proxy
is visible either way.

The adaptive loop now drives both tiers and no longer falls back to
freeze_stale_proxies; that function stays in place for the default fixed
threshold path, which is unchanged.

7 new tests: a single silent proxy convicted, a whole-rack proxy failure held
back, the stall guard suppressing proxy conviction, conviction switchable off
while damage is still reported, a frozen proxy counting as damage without being
reconvicted, proxies never being convicted for a restart, and the two tiers not
sharing a heartbeat distribution.
@bjmeetsfo
bjmeetsfo requested a review from superhaiou as a code owner August 19, 2026 22:54
@matrixarkai
matrixarkai merged commit aa597f5 into main Aug 19, 2026
4 of 5 checks passed
@matrixarkai
matrixarkai deleted the oss/meta-proxy-adaptive-detection branch August 19, 2026 23:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants