Skip to content

feat(shredding): add shared-shredding map placement policies#384

Open
zjw1111 wants to merge 4 commits into
alibaba:mainfrom
zjw1111:codex/extend-map-column-allocator
Open

feat(shredding): add shared-shredding map placement policies#384
zjw1111 wants to merge 4 commits into
alibaba:mainfrom
zjw1111:codex/extend-map-column-allocator

Conversation

@zjw1111

@zjw1111 zjw1111 commented Jun 26, 2026

Copy link
Copy Markdown
Collaborator

Purpose

Linked issue: #342

Add shared-shredding map column placement policies for the write path:

  • Replace the previous policy-state abstraction with allocator implementations for plain, sequential, and LRU placement.
  • Add per-column option fields.<column>.map.shared-shredding.column-placement-policy, defaulting to lru.
  • Keep K adaptation state in MapSharedShreddingContext and let MapSharedShreddingBatchConverter read placement policy from CoreOptions when creating per-column allocators.

Tests

  • cmake --build build --target paimon-common-test -j64
  • cmake --build build --target paimon-core-test -j64
  • env LSAN_OPTIONS=detect_leaks=0 build/debug/paimon-common-test --gtest_filter=*MapSharedShredding*
  • env LSAN_OPTIONS=detect_leaks=0 build/debug/paimon-core-test --gtest_filter=CoreOptionsTest.TestDefaultValue:CoreOptionsTest.TestMapStorageLayout:*SharedShredding*
  • git diff --check
  • git diff --cached --check

API and Format

Adds a core option for selecting the shared-shredding map column placement policy. The physical shared-shredding map column assignment can vary by configured policy; the default policy is lru.

Documentation

No.

Generative AI tooling

Generated-by: OpenAI Codex (GPT-5)

Copilot AI review requested due to automatic review settings June 26, 2026 09:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@zjw1111 zjw1111 force-pushed the codex/extend-map-column-allocator branch 4 times, most recently from 3d6ce68 to 534ca95 Compare June 26, 2026 10:53
@zjw1111 zjw1111 force-pushed the codex/extend-map-column-allocator branch from 534ca95 to fc44b2d Compare June 26, 2026 10:55
Comment thread src/paimon/common/data/shredding/lru_map_shared_shredding_column_allocator.cpp Outdated
Comment thread src/paimon/core/options/map_shared_shredding_column_placement_policy.h Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants