Core, Spark: Fix incorrect partial commit failure count in rewrite data files

[john801205]

Summary

The rewrite data files procedure reported partial commit failures even when all commits succeeded, because it derived the total commit count as min(totalGroupCount, maxCommits). When groups are batched into groupsPerCommit = ceil(totalGroupCount / maxCommits) per commit, the actual number of commits is ceil(totalGroupCount / groupsPerCommit), which can be smaller. The mismatch made totalCommits - succeededCommits positive even with zero failures.

Compute totalCommits as ceil(totalGroupCount / groupsPerCommit) so the failed commit count is correct. Also make succeededCommits an AtomicInteger since it is incremented from multiple rewrite threads.

Issues

• Rewrite data files reports partial commit failure even when all commits succeed #16971

[Copilot]

Pull request overview

Fixes incorrect “partial commit failure” reporting in Spark rewrite_data_files when partial progress is enabled by computing the true number of commits executed (based on groupsPerCommit) rather than min(totalGroupCount, maxCommits). Also makes commit-success tracking thread-safe in the shared commit service and adds regression coverage across supported Spark variants.

Changes:

• Compute totalCommits as ceil(totalGroupCount / groupsPerCommit) to avoid phantom failed-commit counts.
• Make BaseCommitService.succeededCommits an AtomicInteger to safely increment from multiple threads.
• Add regression tests covering the “actual commits < maxCommits” scenario in Spark 3.5, 4.0, and 4.1 test suites.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
core/src/main/java/org/apache/iceberg/actions/BaseCommitService.java	Make succeeded commit counter atomic for concurrent increments during partial-progress commits.
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java	Correct total commit counting logic used to infer failed commits.
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java	Correct total commit counting logic used to infer failed commits.
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java	Correct total commit counting logic used to infer failed commits.
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteDataFilesAction.java	Add regression test ensuring no false failures when actual commits are fewer than maxCommits.
spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteDataFilesAction.java	Add regression test ensuring no false failures when actual commits are fewer than maxCommits.
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteDataFilesAction.java	Add regression test ensuring no false failures when actual commits are fewer than maxCommits.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.