Add StringIndex: a generic open-addressed string set#11660
Draft
dougqh wants to merge 6 commits into
Draft
Conversation
TagSet (datadog.trace.util) — a flat, allocation-free, open-addressed string set: Support (static algorithm over raw int[] hashes / String[] names — the hot path, folds to constants when held in static finals), Data (build-time carrier), and a convenience instance wrapper. 2x-oversized (load factor <= 0.5), linear probe, interned-== fast path, 0 = empty sentinel. Consumers attach a parallel payload array indexed by the returned slot. Benchmarks: SetBenchmark (membership vs HashSet/array/sortedArray/treeSet) and KeyOfBenchmark (name->id resolution vs HashMap and a hand-written switch). TagSetTest covers hashing, probing/wraparound, table-full, and parallel payload. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dougqh
added
comp: core
This comment has been minimized.
This comment has been minimized.
…hmark, not a generic-set one) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…hmark Adds get_tagSetMap / get_tagSetMap_sameKey as a build-once, read-only map option in the map menu: keys in a TagSet, values in a parallel int[] (no boxing), get is Support.indexOf plus one array load. Fastest get in the benchmark and the tightest error bars; results captured in the javadoc. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The shared static sharedLookupIndex was written by every thread under @threads(8), so its cache-line ping-pong capped the get_* benchmarks near a ~1.4B ops/s contention ceiling -- the same artifact SetBenchmark fixed by moving its rotation counter to per-thread @State(Scope.Thread). Apply the same fix here; off the ceiling the get_* numbers rise and the differences between map options stop being compressed. Replaces the two stale, unlabeled Java-21 result blocks (and the interim contention-capped get block) with a single env-stamped table from one run on a modern, representative JVM (Apple M1 Max, macOS 26.4.1, Zulu 17.0.7+7-LTS, 8 threads, 2 forks). Header conclusions re-checked against these numbers: the fixed TagSet map leads HashMap ~30%/~50% (rotating/same-key), forEach matches the fastest map iterators, and create's HashMap-vs-LinkedHashMap edge is ~1.4x. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The structure is more general than tags: a flat, open-addressed String set whose indexOf returns a stable dense slot for parallel value arrays. "Tag" was just one prospective consumer. Renames the class, test, and benchmark references (Support/Data nested types and the SetBenchmark / map-benchmark usages); leaves the unrelated propagation TagSetChanges in dd-trace-core alone. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
dougqh
changed the title
…edMapBenchmark StringIndex's benchmark integration is moving to the dedicated benchmark PRs (set overhaul #11721, map overhaul #11679) and will be folded in there later. Revert both benchmark files to master so this PR is purely the StringIndex data structure + tests. Avoids the #11679/#11721 deletions-vs-edits conflicts too. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
StringIndex(datadog.trace.util, alongside the customHashtable) — a flat, allocation-free, open-addressed string set / index:Support— the static algorithm over rawint[] hashes/String[] names. Held instatic finalfields the refs fold to constants (the hot path).Data— a build-time carrier{int[] hashes, String[] names}(pull into your own fields).of/contains/indexOf).2×-oversized (load factor ≤ 0.5), linear probe + wraparound, hash gates
equals, interned==fast path,0= empty sentinel. Generic — no payload baked in; it just knows names. The headline capability isindexOf, which assigns each known string a stable dense slot; consumers attach a parallel array (e.g.long[] ids) indexed by that slot. Membership (contains) falls out asindexOf >= 0.(Renamed from
TagSet— the structure is more general than tags; a fixed name→id / membership index is just one of its uses.)Why / benchmarks
The motivating use is a fast, declared name→id table — a "pit of success" alternative to hand-rolled string
switches and per-name caches — but the structure is general. Shipped with benchmarks (Apple M1 Max, 8 threads, Java 8):SetBenchmark(membership):StringIndex≈HashSet(~2.0–2.25B ops/s), ~2.5–3×array/sortedArray/treeSet; the staticSupportpath ~12% over the instance and ~12% overHashSeton hits, with much tighter variance.KeyOfBenchmark(name→id):StringIndex≈ 2× a hand-written stringswitch(at 16 cases; the gap widens at scale) and ~20% overHashMap.StringIndexTestcovers hashing/zero-sentinel, probe + wraparound, table-full, and the parallel-payload usage.Follow-ups
StringIndexvsHashSetacross sizes) will land in this PR — flat parallel arrays should beatHashSet's per-elementNodeobjects, especially for small fixed sets. (Footprint, not alloc-rate — different axis.)indexOf→parallel-array ergonomics:TagInterceptor's hot-path tag-nameswitch.🤖 Generated with Claude Code