meta: treat server locations as a hierarchy, not an opaque string - #73
Merged
Conversation
A server's location is a plain String the metaserver only ever compares for exact equality. Real deployments write a path into it - us-east/dc1/az1/rack7 - and the metaserver reads that as one token, which costs correctness twice. Replica spread happens at the wrong granularity. build_shards refuses to put two replicas of a shard in the same location, comparing whole strings. us-east/dc1/az1/rack1 and us-east/dc1/az1/rack2 are different strings, so both are accepted, and the shard ends up with two replicas inside one availability unit. Losing that unit loses both. The check looks like it is spreading across failure domains while it is only spreading across the deepest one. The new test fails on the old code with both replicas in us-east/dc1/az1. A table can only be pinned to exactly one location. preferred_location is matched by string equality, so "keep this table in dc1" is not expressible: an operator can name one rack, or nothing. meta/location.rs parses the string into its slash-separated levels, coarsest first, and supplies the two comparisons those cases need. belongs_to is prefix containment, so a preference naming fewer levels matches every location beneath it. shared_prefix_len measures how much of a failure domain two locations have in common, which is what lets replica placement prefer the widest separation available rather than merely a different leaf. build_shards now walks a separation ladder: try to place each replica in a different top-level domain first, and only narrow when nothing qualifies, down to a different leaf and finally the existing unconstrained pass. Two identical locations are the same failure domain at every granularity and conflict however weak the requirement, which is what preserves today's behaviour for flat tags. Placement eligibility in the rebalancer is now hierarchical too, so a table pinned to us-east/dc1 accepts any server beneath it, a deeper preference still pins to one rack, and a preference no live server matches still falls back to anywhere rather than stranding the shard. A location with no separator has a single level, so a deployment using flat tags behaves exactly as it does today: different strings differ at level 0, and neither is a prefix of the other. Servers that declare no location at all never block placement. This generalises the reference's fixed four-field location - region, datacenter, availability unit, tag - to a path of any depth. The reference compares its last field exactly and the earlier ones only when the pattern sets them; a variable-depth path expresses the same intent as plain prefix matching without forcing every deployment onto exactly four levels. 18 new tests: parsing and separator tolerance, flat tags staying flat, prefix containment in both directions, an empty pattern meaning anywhere, shared-prefix measurement, the separation ladder and its ordering, identical locations never separating, two racks in one availability unit not separating at that level, a preference naming a whole datacenter, a shard outside its datacenter being pulled back, a deeper preference still pinning to one rack, the empty-preference fallback, and the end-to-end topology test above.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
A server's
locationis a plainStringthe metaserver only ever compares for exact equality. Real deployments write a path into it —us-east/dc1/az1/rack7— and the metaserver reads that as one opaque token. That costs correctness twice.Replica spread happens at the wrong granularity.
build_shardsrefuses to put two replicas of a shard in the samelocation, comparing whole strings.us-east/dc1/az1/rack1andus-east/dc1/az1/rack2are different strings, so both are accepted — and the shard ends up with two replicas inside one availability unit. Losing that unit loses both. The check looks like it is spreading across failure domains while it is only spreading across the deepest one.This is measured, not argued: the new end-to-end test fails on current
mainwithA table can only be pinned to exactly one location.
preferred_locationis matched by string equality, so "keep this table in dc1" is not expressible — an operator can name one rack, or nothing.What this adds
meta/location.rsparses the string into its/-separated levels, coarsest first, and supplies the two comparisons those cases need:belongs_to— prefix containment. A preference naming fewer levels matches every location beneath it.shared_prefix_len— how much of a failure domain two locations have in common, which is what lets placement prefer the widest separation available rather than merely a different leaf.build_shardsnow walks a separation ladder: place each replica in a different top-level domain if it can, and only narrow when nothing qualifies — down to a different leaf, and finally the existing unconstrained pass. Two identical locations are the same failure domain at every granularity and conflict however weak the requirement, which is what preserves today's behaviour for flat tags.Placement eligibility in the rebalancer is hierarchical too, so a table pinned to
us-east/dc1accepts any server beneath it, a deeper preference still pins to one rack, and a preference no live server matches still falls back to anywhere rather than stranding the shard.Compatibility
A location with no separator has a single level, so a deployment using flat tags (
zone-a,zone-b) behaves exactly as it does today: different strings differ at level 0, and neither is a prefix of the other. Servers that declare no location at all never block placement. No wire type changes — the field is still aString, just interpreted.Relationship to the reference
This generalises the reference's fixed four-field location (region / datacenter / availability unit / tag) to a path of any depth. The reference compares its last field exactly and the earlier ones only when the pattern sets them; a variable-depth path expresses the same intent as plain prefix matching, without forcing every deployment onto exactly four levels.
Tests
18 new tests: parsing and separator tolerance, flat tags staying flat, prefix containment in both directions, an empty pattern meaning anywhere, shared-prefix measurement, the separation ladder and its ordering, identical locations never separating, two racks in one availability unit not separating at that level, a preference naming a whole datacenter, a shard outside its datacenter being pulled back, a deeper preference still pinning to one rack, the empty-preference fallback, and the end-to-end topology test above.
Verification:
cargo test -p temporalstore-rust --lib meta::location— 14 passed, 0 failed.cargo test -p temporalstore-rust --lib meta::placement_rebalance— 16 passed, 0 failed.cargo test -p temporalstore-rust --lib meta— 181 passed, 0 failed.cargo test -p temporalstore-rust --bin metaserver— 18 passed, 0 failed.partitioning.rsand fails, confirming it is a genuine regression test rather than a tautology.