Sai prefixv2#4909
Conversation
| * Depth is 1-based on the key bytes (the first byte is depth 1); the empty prefix (depth 0, the root) is never | ||
| * accumulated. A null policy is equivalent to {@link #putSingleton(ByteComparable, Object, UpsertTransformer, boolean)}. | ||
| */ | ||
| public <R> void putSingleton(ByteComparable key, |
There was a problem hiding this comment.
TODO: avoid code duplication. probably we could pass a new arg to putSingleton() and/or putRecursive.
We might need to support both the flows as well.
| totalPostings++; | ||
| } | ||
|
|
||
| if (count > 0) |
There was a problem hiding this comment.
If it writes to the disk, we are technically doing two different IOs (one for exact and other for Prefix), this can be avoided
| } | ||
|
|
||
| this.blockSize = input.readVInt(); | ||
| //TODO This should need to change because we can potentially end up with postings of more than Integer.MAX_VALUE? |
There was a problem hiding this comment.
should we change it to long since we will have more postings with prefix?
| { | ||
| ByteBuffer prefixValue = expression.lower().value.encoded; | ||
| ByteComparable start = v -> index.termType().asComparableBytes(prefixValue, v); | ||
| ByteBuffer successor = prefixSuccessor(prefixValue); |
There was a problem hiding this comment.
successor is used to pass it to the in-memory subtrie() API. do we actually need the successor here?
| if (indexTermType.isLiteral() && literalPrefixEnabled) | ||
| { | ||
| int skip = CassandraRelevantProperties.SAI_POSTINGS_SKIP.getInt(); | ||
| SegmentTrieBuffer buffer = new SegmentTrieBuffer(depth -> depth % skip == 0); |
There was a problem hiding this comment.
TODO: avoid using segmentriebuffer instead try to do it in place for memtableIndexwriter
| { | ||
| ByteBuffer prefixBuffer = expression.lower().value.encoded; | ||
| ByteComparable lowerBound = asComparableBytes(prefixBuffer); | ||
| ByteBuffer successor = prefixSuccessor(prefixBuffer); |
There was a problem hiding this comment.
revisit if we really need prefixSuccessor
| { | ||
| int contentIndex = ~node; | ||
| setContent(contentIndex, transformer.apply(getContent(contentIndex), value)); | ||
| setContent(contentIndex, combine(transformer, getContent(contentIndex), value, intermediate)); |
There was a problem hiding this comment.
should we overload transformer.apply() with an extra arg instead of applyIntermeditate?
|
|
||
| // Block range [startBlock, endBlock) this reader is scoped to, and the number of postings to read. | ||
| // For a full (V1) posting list these are 0, numBlocks and numPostings respectively. | ||
| private final int startBlock; |
There was a problem hiding this comment.
check if the design scalable to suffix as well.
|
|
||
| int[] prefixRows = null; | ||
| if (writePrefixSection && prefixCount > 0) | ||
| prefixRows = drainSortedInts(postings, prefixCount); |
There was a problem hiding this comment.
TODO: see if we can guarentee both exact list and prefix list are already sorted
| maxSSTableRowId = Math.max(maxSSTableRowId, maxSegmentRowID); | ||
| if (isV2) | ||
| { | ||
| for (int i = V2_HEADER_SIZE; i < list.size(); i++) |
There was a problem hiding this comment.
TODO: if the exactPostingsList and prefixPostingsList are sorted seperately, then we don't need the loop
| /** Primary keys whose indexed term exactly equals this trie node's key. Never null. */ | ||
| public PrimaryKeys exact() | ||
| { | ||
| return exact; |
There was a problem hiding this comment.
should we send an iterator instead?
1 participant
Thanks for sending a pull request! Here are some tips if you're new here:
Commit messages should follow the following format:
The Cassandra Jira