Skip to content

[python] Advance overwrite changes cache on retry#8368

Open
JingsongLi wants to merge 1 commit into
apache:masterfrom
JingsongLi:codex/python-commit-retry-apply-delta
Open

[python] Advance overwrite changes cache on retry#8368
JingsongLi wants to merge 1 commit into
apache:masterfrom
JingsongLi:codex/python-commit-retry-apply-delta

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Jun 27, 2026

Copy link
Copy Markdown
Contributor

Summary

Follow up #8358 by extending the overwrite-specific retry cache. OverwriteChangesProvider now advances its cached target-partition state with intervening raw delta entries, so overwrite retries do not rebuild the DELETE list with another full scan when those deltas can be applied safely.

Relates to #8359.

Changes

  • Apply intervening ADD/DELETE delta entries to the cached overwrite target state with FileEntry.merge_entries.
  • Reuse the overwrite changes cache across target-partition appends, concurrent overwrites, and whole-table overwrite retries.
  • Fall back to the existing full scan path if a snapshot is missing, a delta manifest cannot be read, or applying the delta fails.
  • Update overwrite cache regression tests to cover append deltas, overwrite deltas, whole-table overwrite, and dynamic-partition overwrite.

Testing

  • PYTHONPATH=paimon-python python -m pytest paimon-python/pypaimon/tests/overwrite_changes_cache_test.py -q
  • PYTHONPATH=paimon-python python -m pytest paimon-python/pypaimon/tests/overwrite_commit_conflict_test.py -q
  • PYTHONPATH=paimon-python python -m pytest paimon-python/pypaimon/tests/file_store_commit_test.py -q
  • PYTHONPATH=paimon-python python -m pytest paimon-python/pypaimon/tests/partition_predicate_test.py -q
  • PYTHONPATH=paimon-python python -m pytest paimon-python/pypaimon/tests/write/conflict_detection_test.py -q
  • PYTHONPATH=paimon-python python -m pytest paimon-python/pypaimon/tests/table_commit_test.py -q
  • PYTHONPATH=paimon-python python -m pytest paimon-python/pypaimon/tests/manifest/manifest_manager_test.py -q
  • PYTHONPATH=paimon-python python -m py_compile paimon-python/pypaimon/write/commit/overwrite_changes_provider.py paimon-python/pypaimon/tests/overwrite_changes_cache_test.py
  • git diff --check

Notes

The generic commit retry base-state reuse is already handled by #8358. This PR is intentionally scoped to overwrite commit-entry construction and only changes OverwriteChangesProvider plus its cache tests.

@JingsongLi JingsongLi force-pushed the codex/python-commit-retry-apply-delta branch 2 times, most recently from abe516e to 6cd6612 Compare June 27, 2026 16:10
@JingsongLi JingsongLi force-pushed the codex/python-commit-retry-apply-delta branch from 6cd6612 to 39dabb0 Compare June 27, 2026 16:22
@JingsongLi JingsongLi changed the title [python] Apply incremental commit retry changes [python] Apply overwrite delta changes on retry Jun 27, 2026
@JingsongLi JingsongLi changed the title [python] Apply overwrite delta changes on retry [python] Advance overwrite changes cache on retry Jun 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant