Skip to content

fix(load): use buffered input when splitting TsFiles - #18388

Open
luoluoyuyu wants to merge 3 commits into
apache:masterfrom
luoluoyuyu:codex/load-buffered-tsfile-input
Open

fix(load): use buffered input when splitting TsFiles#18388
luoluoyuyu wants to merge 3 commits into
apache:masterfrom
luoluoyuyu:codex/load-buffered-tsfile-input

Conversation

@luoluoyuyu

Copy link
Copy Markdown
Member

Description

As the title said


This PR has:

  • been self-reviewed.
    • concurrent read
    • concurrent write
    • concurrent read and write
  • added documentation for new or modified features or behaviors.
  • added Javadocs for most classes and all non-trivial methods.
  • added or updated version, license, or notice information
  • added comments explaining the "why" and the intent of the code wherever would not be obvious
    for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold
    for code coverage.
  • added integration tests.
  • been tested in a test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR

throws IOException, LoadFileException, IllegalStateException {
try (TsFileSequenceReader reader = new TsFileSequenceReader(tsFile.getAbsolutePath())) {
try (TsFileSequenceReader reader =
new TsFileSequenceReader(new BufferedTsFileInput(tsFile.toPath()))) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This overload skips TsFileSequenceReader.loadFileVersion(). The TsFileSequenceReader(TsFileInput) constructor only loads the metadata size, so a v3 file keeps the default v4 DeserializeConfig. TsFileSplitter.checkMagic() explicitly accepts v3, but getAllTimeseriesMetadata(true) then fails while parsing its metadata.

I reproduced this with iotdb-core/datanode/src/test/resources/v3tsfile/compaction-test-tsfile: the original string constructor reads 2 devices, whereas this constructor throws IllegalArgumentException: Invalid input: 114 from MetadataIndexNodeType.deserialize(). The four-argument buffered constructor works:

new TsFileSequenceReader(
    new BufferedTsFileInput(tsFile.toPath()), true, false, null)

Please preserve the version initialization (for example by using that overload) and add a v3 splitter regression test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants