Skip to content

meta: treat server locations as a hierarchy, not an opaque string - #73

Merged
matrixarkai merged 1 commit into
mainfrom
oss/meta-hierarchical-locations
Aug 20, 2026
Merged

meta: treat server locations as a hierarchy, not an opaque string#73
matrixarkai merged 1 commit into
mainfrom
oss/meta-hierarchical-locations

Conversation

@bjmeetsfo

Copy link
Copy Markdown
Collaborator

The problem

A server's location is a plain String the metaserver only ever compares for exact equality. Real deployments write a path into it — us-east/dc1/az1/rack7 — and the metaserver reads that as one opaque token. That costs correctness twice.

Replica spread happens at the wrong granularity. build_shards refuses to put two replicas of a shard in the same location, comparing whole strings. us-east/dc1/az1/rack1 and us-east/dc1/az1/rack2 are different strings, so both are accepted — and the shard ends up with two replicas inside one availability unit. Losing that unit loses both. The check looks like it is spreading across failure domains while it is only spreading across the deepest one.

This is measured, not argued: the new end-to-end test fails on current main with

both replicas landed in the same availability unit: {"us-east/dc1/az1"}

A table can only be pinned to exactly one location. preferred_location is matched by string equality, so "keep this table in dc1" is not expressible — an operator can name one rack, or nothing.

What this adds

meta/location.rs parses the string into its /-separated levels, coarsest first, and supplies the two comparisons those cases need:

  • belongs_to — prefix containment. A preference naming fewer levels matches every location beneath it.
  • shared_prefix_len — how much of a failure domain two locations have in common, which is what lets placement prefer the widest separation available rather than merely a different leaf.

build_shards now walks a separation ladder: place each replica in a different top-level domain if it can, and only narrow when nothing qualifies — down to a different leaf, and finally the existing unconstrained pass. Two identical locations are the same failure domain at every granularity and conflict however weak the requirement, which is what preserves today's behaviour for flat tags.

Placement eligibility in the rebalancer is hierarchical too, so a table pinned to us-east/dc1 accepts any server beneath it, a deeper preference still pins to one rack, and a preference no live server matches still falls back to anywhere rather than stranding the shard.

Compatibility

A location with no separator has a single level, so a deployment using flat tags (zone-a, zone-b) behaves exactly as it does today: different strings differ at level 0, and neither is a prefix of the other. Servers that declare no location at all never block placement. No wire type changes — the field is still a String, just interpreted.

Relationship to the reference

This generalises the reference's fixed four-field location (region / datacenter / availability unit / tag) to a path of any depth. The reference compares its last field exactly and the earlier ones only when the pattern sets them; a variable-depth path expresses the same intent as plain prefix matching, without forcing every deployment onto exactly four levels.

Tests

18 new tests: parsing and separator tolerance, flat tags staying flat, prefix containment in both directions, an empty pattern meaning anywhere, shared-prefix measurement, the separation ladder and its ordering, identical locations never separating, two racks in one availability unit not separating at that level, a preference naming a whole datacenter, a shard outside its datacenter being pulled back, a deeper preference still pinning to one rack, the empty-preference fallback, and the end-to-end topology test above.

Verification:

  • cargo test -p temporalstore-rust --lib meta::location — 14 passed, 0 failed.
  • cargo test -p temporalstore-rust --lib meta::placement_rebalance — 16 passed, 0 failed.
  • cargo test -p temporalstore-rust --lib meta181 passed, 0 failed.
  • cargo test -p temporalstore-rust --bin metaserver — 18 passed, 0 failed.
  • The spread test was run against unmodified partitioning.rs and fails, confirming it is a genuine regression test rather than a tautology.

A server's location is a plain String the metaserver only ever compares for
exact equality. Real deployments write a path into it - us-east/dc1/az1/rack7 -
and the metaserver reads that as one token, which costs correctness twice.

Replica spread happens at the wrong granularity. build_shards refuses to put two
replicas of a shard in the same location, comparing whole strings.
us-east/dc1/az1/rack1 and us-east/dc1/az1/rack2 are different strings, so both
are accepted, and the shard ends up with two replicas inside one availability
unit. Losing that unit loses both. The check looks like it is spreading across
failure domains while it is only spreading across the deepest one. The new test
fails on the old code with both replicas in us-east/dc1/az1.

A table can only be pinned to exactly one location. preferred_location is
matched by string equality, so "keep this table in dc1" is not expressible: an
operator can name one rack, or nothing.

meta/location.rs parses the string into its slash-separated levels, coarsest
first, and supplies the two comparisons those cases need. belongs_to is prefix
containment, so a preference naming fewer levels matches every location beneath
it. shared_prefix_len measures how much of a failure domain two locations have
in common, which is what lets replica placement prefer the widest separation
available rather than merely a different leaf.

build_shards now walks a separation ladder: try to place each replica in a
different top-level domain first, and only narrow when nothing qualifies, down
to a different leaf and finally the existing unconstrained pass. Two identical
locations are the same failure domain at every granularity and conflict however
weak the requirement, which is what preserves today's behaviour for flat tags.

Placement eligibility in the rebalancer is now hierarchical too, so a table
pinned to us-east/dc1 accepts any server beneath it, a deeper preference still
pins to one rack, and a preference no live server matches still falls back to
anywhere rather than stranding the shard.

A location with no separator has a single level, so a deployment using flat tags
behaves exactly as it does today: different strings differ at level 0, and
neither is a prefix of the other. Servers that declare no location at all never
block placement.

This generalises the reference's fixed four-field location - region,
datacenter, availability unit, tag - to a path of any depth. The reference
compares its last field exactly and the earlier ones only when the pattern sets
them; a variable-depth path expresses the same intent as plain prefix matching
without forcing every deployment onto exactly four levels.

18 new tests: parsing and separator tolerance, flat tags staying flat, prefix
containment in both directions, an empty pattern meaning anywhere, shared-prefix
measurement, the separation ladder and its ordering, identical locations never
separating, two racks in one availability unit not separating at that level, a
preference naming a whole datacenter, a shard outside its datacenter being
pulled back, a deeper preference still pinning to one rack, the empty-preference
fallback, and the end-to-end topology test above.
@bjmeetsfo
bjmeetsfo requested a review from superhaiou as a code owner August 19, 2026 23:43
@matrixarkai
matrixarkai merged commit ab7455a into main Aug 20, 2026
4 of 5 checks passed
@matrixarkai
matrixarkai deleted the oss/meta-hierarchical-locations branch August 20, 2026 00:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants