meta: give proxies the same adaptive detection and safe mode as datanodes - #71
Merged
Merged
Conversation
…odes The adaptive failure detector deliberately left proxies on the fixed stale_after_ms threshold, on the grounds that freezing a proxy is cheap because it moves no data. That reasoning covers one of the three problems the detector was built for and misses the two that matter here. A proxy freeze being individually cheap says nothing about the correlated case. The fixed threshold is measured against wall clock, so a metaserver stall makes every proxy cross it at the same instant, and a rack fault makes every proxy behind that rack look stale together. The proxies are the routing tier: freezing all of them is a total outage of the serving path. The blast radius of a correlated proxy failure is larger than the datanode equivalent, not smaller, which makes the proxies the tier that needs the guard most. Proxies now run the same round the datanodes do: phi-accrual detection against each proxy's own learned heartbeat cadence, the stall guard that suppresses conviction after the detector itself pauses, and the per-location safe mode that holds back conviction when too much of one location is failing at once. The round body is factored into one shared function over a ConvictionSubject, so the two tiers cannot drift apart - a change to detection applies to both by construction. A proxy carries no boot-time anchor, so the reboot path never fires for one; only silence convicts a proxy. Each tier gets its own detector instance. Datanode heartbeat cadence says nothing about proxy cadence, so sharing one would judge each against the other's rhythm, and separate stall clocks mean a pause is judged against the tier it actually affected. ConvictionPolicy::convict_proxies (on by default, TS_META_CONVICT_PROXIES to disable) lets an operator hold the routing tier steady while datanode conviction runs. With it off the damage is still assessed and reported, so a failing proxy is visible either way. The adaptive loop now drives both tiers and no longer falls back to freeze_stale_proxies; that function stays in place for the default fixed threshold path, which is unchanged. 7 new tests: a single silent proxy convicted, a whole-rack proxy failure held back, the stall guard suppressing proxy conviction, conviction switchable off while damage is still reported, a frozen proxy counting as damage without being reconvicted, proxies never being convicted for a restart, and the two tiers not sharing a heartbeat distribution.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
#61 deliberately left proxies on the fixed
stale_after_msthreshold, arguing that freezing a proxy is cheap because it moves no data.That covers one of the three problems the detector was built for and misses the two that matter here. A proxy freeze being individually cheap says nothing about the correlated case:
The proxies are the routing tier. Freezing all of them is a total outage of the serving path. The blast radius of a correlated proxy failure is larger than the datanode equivalent, not smaller — which makes proxies the tier that needs the guard most, not the one that can go without it.
The reference metaserver runs phi detection and per-tag safe mode for proxies exactly as it does for servers.
What this does
Proxies now run the same round the datanodes do: phi-accrual detection against each proxy's own learned heartbeat cadence, the stall guard that suppresses conviction after the detector itself pauses, and the per-location safe mode that holds back conviction when too much of one location is failing at once.
The round body is factored into one shared function over a
ConvictionSubject, so the two tiers cannot drift apart — a change to detection applies to both by construction.Each tier gets its own detector instance. Datanode heartbeat cadence says nothing about proxy cadence, so sharing one would judge each against the other's rhythm; separate stall clocks also mean a pause is judged against the tier it actually affected. A test asserts the two do not share distributions.
A proxy carries no boot-time anchor, so the reboot path (#64) never fires for one — only silence convicts a proxy.
TS_META_CONVICT_PROXIES1With it off, damage is still assessed and reported, so a failing proxy stays visible either way. It is a separate knob from
TS_META_CONVICT_ENABLEDbecause the two tiers fail for different reasons and an operator may want to hold the routing tier steady while datanode conviction runs.The adaptive loop now drives both tiers and no longer falls back to
freeze_stale_proxies; that function stays in place for the default fixed-threshold path, which is unchanged.Tests
7 new tests: a single silent proxy convicted, a whole-rack proxy failure held back, the stall guard suppressing proxy conviction, conviction switchable off while damage is still reported, a frozen proxy counting as damage without being reconvicted, proxies never being convicted for a restart, and the two tiers not sharing a heartbeat distribution.
Verification:
cargo test -p temporalstore-rust --lib meta::failure_detector— 28 passed, 0 failed.cargo test -p temporalstore-rust --lib meta— 159 passed, 0 failed. (One earlier run showed the two known-flakyproxy::testsfailures that also fail on pristinemain; they passed on re-run.)cargo test -p temporalstore-rust --bin metaserver— 18 passed, 0 failed.cargo build -p temporalstore-rust --bin metaserver— clean, no new warnings.Still off by default behind
TS_META_ADAPTIVE_FAILURE_DETECTOR.