[python] Roll manifest files by target size during commit and compaction by XiaoHongbo-Hope · Pull Request #8373 · apache/paimon

[python] Split manifest files by target size during commit and compaction

XiaoHongbo-Hope

committed

[python] Fix rolling_write: clean up on partial failure, remove double serialization

XiaoHongbo-Hope

committed

[python] Use size-aware adaptive rolling instead of avg-based chunk splitting

XiaoHongbo-Hope

committed

[python] Replace fragile name parsing with explicit name_prefix parameter

XiaoHongbo-Hope

committed

[python] Fix flake8 E128 continuation line indent in manifest tests

XiaoHongbo-Hope

committed

[python] Update file_store_commit_test for renamed _write_manifest_files

XiaoHongbo-Hope

committed

[python] Use sample-based size estimation in rolling_write

XiaoHongbo-Hope

committed

[python] Track rolling manifest files in new_manifest_files_for_abort

XiaoHongbo-Hope

committed

[python] Use streaming rolling write aligned with Java RollingFileWriter

XiaoHongbo-Hope

committed

[python] Use sync_interval for per-record size tracking in rolling write

XiaoHongbo-Hope

committed

[python] Extract sync_interval magic number to _AVRO_SYNC_INTERVAL constant

XiaoHongbo-Hope

committed

[python] Allow merger to split oversized single-file manifest groups

XiaoHongbo-Hope

committed

[python] Revert merger single-file split to match Java behavior

XiaoHongbo-Hope

committed

[python] Clean up manifest files via new_manifest_files_for_abort on preparation failure

XiaoHongbo-Hope

committed

[python] Pass known file size to _build_meta to avoid remote HEAD requests

XiaoHongbo-Hope

committed

[python] Track written files separately for rollback in rolling_write

XiaoHongbo-Hope

committed

[python] Add read-back assertion for rolling manifest files

XiaoHongbo-Hope

committed

[python] Remove extra blank line between test classes

XiaoHongbo-Hope

committed

[python] Remove redundant manifest file cleanup in except block

XiaoHongbo-Hope

committed

[python] Align cleanup with Java CommitCleaner two-step structure

XiaoHongbo-Hope

committed

[python] Add known metas fallback to _clean_up_reuse_tmp_manifests

XiaoHongbo-Hope

committed

[python] Guard cleanup with try/except, simplify merge new_files, remove dead code

XiaoHongbo-Hope

committed

[python] Fix unbound merge_new_files and remove unused merge_after_manifests

XiaoHongbo-Hope

committed

[python] Remove trailing blank line at EOF

XiaoHongbo-Hope

committed

[python] Remove known metas fallback to match Java CommitCleaner

XiaoHongbo-Hope

committed

[python] Fix flake8 E128 indent in cleanup method signatures

XiaoHongbo-Hope

committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[python] Roll manifest files by target size during commit and compaction#8373

[python] Roll manifest files by target size during commit and compaction#8373
JingsongLi merged 26 commits into
apache:masterfrom
XiaoHongbo-Hope:manifest_write_fix

Commits on Jun 28, 2026

[python] Split manifest files by target size during commit and compaction

[python] Fix rolling_write: clean up on partial failure, remove double serialization

[python] Use size-aware adaptive rolling instead of avg-based chunk splitting

[python] Replace fragile name parsing with explicit name_prefix parameter

[python] Fix flake8 E128 continuation line indent in manifest tests

[python] Update file_store_commit_test for renamed _write_manifest_files

[python] Use sample-based size estimation in rolling_write

[python] Track rolling manifest files in new_manifest_files_for_abort

[python] Use streaming rolling write aligned with Java RollingFileWriter

[python] Use sync_interval for per-record size tracking in rolling write

[python] Extract sync_interval magic number to _AVRO_SYNC_INTERVAL constant

[python] Allow merger to split oversized single-file manifest groups

[python] Revert merger single-file split to match Java behavior

[python] Clean up manifest files via new_manifest_files_for_abort on preparation failure

[python] Pass known file size to _build_meta to avoid remote HEAD requests

[python] Track written files separately for rollback in rolling_write

[python] Add read-back assertion for rolling manifest files

[python] Remove extra blank line between test classes

[python] Remove redundant manifest file cleanup in except block

[python] Align cleanup with Java CommitCleaner two-step structure

[python] Add known metas fallback to _clean_up_reuse_tmp_manifests

[python] Guard cleanup with try/except, simplify merge new_files, remove dead code

[python] Fix unbound merge_new_files and remove unused merge_after_manifests

[python] Remove trailing blank line at EOF

[python] Remove known metas fallback to match Java CommitCleaner

[python] Fix flake8 E128 indent in cleanup method signatures