[To dev/1.3] clone partial columns of aligned tvlist for query - #18394
Open
shuwenwei wants to merge 12 commits into
Open
[To dev/1.3] clone partial columns of aligned tvlist for query#18394shuwenwei wants to merge 12 commits into
shuwenwei wants to merge 12 commits into
Conversation
# Conflicts: # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/datastructure/AlignedTVList.java
…l clone Re-fetch and re-verify the working TVList under the memChunk lock after acquiring its queryListLock. The clone, column move and working-list swap now happen in the same memChunk critical section, so a concurrent query can never observe a working TVList whose uncloned columns were already moved away, and the fast paths never read a detached list.
… calc refreshArrayMemCostWithoutIndex() was invoked before the values field was initialized, but alignedTvListArrayMemCost(Set) now dereferences values.get(column) to skip moved columns, so construction of any aligned TVList threw NPE. Initialize values before computing the array mem cost.
…b into partial_tvlist_clone_dev_1.3
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## dev/1.3 #18394 +/- ##
===========================================
Coverage 44.29% 44.30%
Complexity 198 198
===========================================
Files 3634 3634
Lines 245511 245718 +207
Branches 30045 30118 +73
===========================================
+ Hits 108746 108859 +113
- Misses 136765 136859 +94 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
For wide aligned devices, a query that only touches a few columns currently
clones the entire aligned TVList (all columns) in the working memtable, which
is very expensive in both memory and time. This PR makes the query-side clone
copy only the accessed columns and move the remaining uncloned columns by
reference into the clone, so partial-column queries no longer duplicate the
whole aligned device.
Changes
AlignedTVList: addclone(Set<Integer>),cloneColumnDataToandmoveUnclonedColumnsToso the clone keeps only the accessed columns and theuncloned columns are transferred by reference into the new working list.
ResourceByPathUtils.prepareTvListMapForQuery: pass the accessed columnindex list into the clone path; re-fetch and re-verify the working TVList
under the memChunk lock after acquiring its
queryListLock, and performclone + column move + working-list swap in the same memChunk critical
section so a concurrent query never observes a detached list whose columns
were already moved away.
FragmentInstanceContext: track column-level access for AlignedTVList.Tests
AlignedTVListTest: partial clone keeps only the accessed columns and movesthe uncloned ones by reference.
FragmentInstanceExecutionTest: querying partial columns of an aligneddevice returns correct results.
This PR has: