Skip to content

feat(array): add sorted membership mask - #9552

Open
aunjgr wants to merge 1 commit into
vortex-data:developfrom
aunjgr:benthoscape-sorted-binary-set
Open

feat(array): add sorted membership mask#9552
aunjgr wants to merge 1 commit into
vortex-data:developfrom
aunjgr:benthoscape-sorted-binary-set

Conversation

@aunjgr

@aunjgr aunjgr commented Aug 22, 2026

Copy link
Copy Markdown

Summary

Add vectorized exact-membership masks for validated sorted arrays. Engine integrations can now turn
an ordering proof into bounded binary-search range narrowing plus one linear merge, without scalar
extraction, per-row searches, copied variable-width values, or a temporary hash table.

The operation is useful for sorted set predicates and joins, delete-key application, sorted-run
reconciliation, set operations, dictionary-domain filtering, and exact runtime filters.

Changes

  • Add SortedArray, SortedOrder, SortedDirection, SortedNulls, and NullEquality.
  • Add sorted_membership_mask for Bool, Primitive, Decimal, Binary, and UTF-8 arrays.
  • Canonicalize and validate members once, then retain their canonical representation and validity.
  • Validate each probe batch while constructing its result mask.
  • Narrow the member interval with two binary searches before merging.
  • Support ascending/descending order, explicit null placement/equality, duplicates, and differing
    decimal physical widths.
  • Preserve Vortex's existing primitive-float total order and bitwise equality.
  • Add unit coverage for correctness, rejection paths, and bounded member comparisons.
  • Extend the existing search_sorted benchmark with native i64 and external BinaryView membership.
  • Validate with the complete vortex-array suite, doctests, pinned-nightly formatting, and strict
    all-target clippy.

Comparative Results

Median times for one 8,192-row probe chunk:

members / hit density narrowed merge full merge per-row binary search hash probe sorted validation hash build
16K / 100% 54.1 us 15.6 us 100.1 us 22.3 us 6.3 us 58.1 us
65K / 25% 38.0 us 22.0 us 87.8 us 17.0 us 24.7 us 240.8 us
1M / 6.25% 35.3 us 105.1 us 115.9 us 18.2 us 381.4 us 5.77 ms

For the large sparse case, narrowing is about 3x faster than rescanning the member set and 3.3x
faster than one binary search per row. A prebuilt hash set remains about 2x faster to probe, while
its measured construction is about 15x slower. This API is therefore an alternative for existing
sorted state and ordered storage, not a universal replacement for hash membership; integrations
should cost both.

API Changes

pub struct SortedArray { /* validated canonical member set */ }

pub fn sorted_membership_mask(
    values: &ArrayRef,
    members: &SortedArray,
    null_equality: NullEquality,
    ctx: &mut ExecutionCtx,
) -> VortexResult<Mask>

AI assistance disclosure: OpenAI Codex was used to help draft the implementation, tests,
benchmark, and PR text. The human author reviewed the result and remains accountable for the
contribution.

Signed-off-by: bRong Njam <longran1989@gmail.com>
@joseph-isaacs

Copy link
Copy Markdown
Contributor

We have sorted-ness stats which could be used here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants