meta: record why a resource was frozen, and let an operator unfreeze it - #75
Open
bjmeetsfo wants to merge 1 commit into
Open
meta: record why a resource was frozen, and let an operator unfreeze it#75bjmeetsfo wants to merge 1 commit into
bjmeetsfo wants to merge 1 commit into
Conversation
Freezing recorded no reason at all. The metaserver could not tell an operator taking a node out for maintenance from the failure detector convicting one that stopped answering, even though the two want opposite recovery behaviour. There was also no way out of a freeze. Tables have unfreeze_table; servers and proxies had nothing. The only path back to Normal was for the frozen resource to re-register itself, and apply_register_server permits that as soon as the freeze cooldown has elapsed. The cooldown defaults to zero, so a convicted node re-registers immediately and returns itself to service: the failure detector's decision is erased by the very node it was about, and conviction is advisory rather than binding. FreezeReason is now recorded on servers and proxies - Operator for an explicit admin request, Unresponsive when the detector convicts for silence, Restarted when reboot detection fires - and reported through list_servers and the topology event stream, so an operator can see why a node is out. The reason is set at every freeze site: the stale-heartbeat sweep and the adaptive detector mark their own verdicts as convictions, and the adaptive path distinguishes a restart from a silence using the plan it already computes. unfreeze_server and unfreeze_proxy return a frozen resource to service, exposed as POST /servers/unfreeze and /proxies/unfreeze. They are always available whatever the freeze reason, because an operator has to be able to overrule the metaserver. Unfreeze is recorded as its own mutation: it was previously the one state change that recorded nothing, so mutation-log recovery would have silently re-frozen the resource, and a test covers that replay. With TS_META_FORBID_SELF_CLEARING_CONVICTION the metaserver refuses a re-registration from a resource it convicted, returning conviction_requires_unfreeze until an operator intervenes. An operator freeze is deliberately not covered: that already has a human in the loop, and the freeze cooldown is the knob for holding a node out. Off by default. The automatic recovery it removes is load-bearing wherever the freeze cooldown is left at zero, so turning it on is a deployment decision. Recording the reason and offering unfreeze are unconditional - neither changes existing behaviour. 7 new tests: the reason recorded for a detector freeze and an operator freeze, a convicted server refused re-registration and readmitted after unfreeze, the default path still recovering automatically, an operator freeze not being locked, a restart locked out under its own reason, a convicted proxy covered the same way, and an unfreeze surviving mutation-log replay.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
Freezing recorded no reason at all. The metaserver could not tell an operator taking a node out for maintenance from the failure detector convicting one that stopped answering — even though the two want opposite recovery behaviour.
There was also no way out of a freeze. Tables have
unfreeze_table; servers and proxies had nothing. The only path back toNormalwas for the frozen resource to re-register itself, andapply_register_serverpermits that as soon as the freeze cooldown has elapsed.The cooldown defaults to zero. So a convicted node re-registers immediately and returns itself to service: the failure detector's decision is erased by the very node it was about, and conviction is advisory rather than binding.
What this adds
FreezeReasonon servers and proxies —Operatorfor an explicit admin request,Unresponsivewhen the detector convicts for silence,Restartedwhen reboot detection (#64) fires. Reported throughlist_serversand the topology event stream, so an operator can see why a node is out. Set at every freeze site: the stale-heartbeat sweep and the adaptive detector mark their own verdicts as convictions, and the adaptive path distinguishes a restart from a silence using the plan it already computes.unfreeze_server/unfreeze_proxy, exposed asPOST /servers/unfreezeandPOST /proxies/unfreeze. Always available whatever the freeze reason, because an operator has to be able to overrule the metaserver.Unfreeze is recorded as its own mutation. It was previously the one state change that recorded nothing, so mutation-log recovery would have silently re-frozen the resource — a test covers that replay.
TS_META_FORBID_SELF_CLEARING_CONVICTIONmakes the metaserver refuse a re-registration from a resource it convicted, returningconviction_requires_unfreezeuntil an operator intervenes.An operator freeze is deliberately not covered by the lock: that already has a human in the loop, and the freeze cooldown is the knob for holding a node out.
Off by default
TS_META_FORBID_SELF_CLEARING_CONVICTION0The automatic recovery this removes is load-bearing wherever the freeze cooldown is left at zero, so turning it on is a deployment decision. Recording the reason and offering unfreeze are unconditional — neither changes existing behaviour.
Tests
7 new tests: the reason recorded for a detector freeze and for an operator freeze, a convicted server refused re-registration and readmitted after unfreeze, the default path still recovering automatically, an operator freeze not being locked, a restart locked out under its own reason, a convicted proxy covered the same way, and an unfreeze surviving mutation-log replay.
Verification:
cargo test -p temporalstore-rust --lib meta::tests— 34 passed, 0 failed.cargo test -p temporalstore-rust --lib meta— 176 passed, 0 failed.cargo test -p temporalstore-rust --bin metaserver— 18 passed, 0 failed.cargo build -p temporalstore-rust --bin metaserver— clean, no new warnings.Independent of #73.