Core, Data, Spark, Flink: Remove of position delete files with row data by dramaticlly · Pull Request #17706 · apache/iceberg

dramaticlly · 2026-08-18T03:12:17Z

Follow up removal of #14045, this removes ability of writing Position Deletes With Row (PDWR) in java reference implementation. https://iceberg.apache.org/spec/#position-delete-files-with-row-data

After the change, the Java implementation will no longer write PDWR delete files. It will only support reading the position and file name columns from previously created delete files or those generated by other engines.

Core

PositionDelete.set(CharSequence, long, R) and getter of row() removed while keep the size() == 3 and the positional get/set so that delete files already containing row data still read back correctly.
Avro, ORC and Parquet buildPositionWriter() now throw UnsupportedOperationException when a rowSchema is set.
rowSchema itself stays on all three DeleteWriteBuilders as it is still required for equality deletes
Avro.PositionAndRowDatumWriter deleted; the PositionDeleteStructWriters in GenericOrcWriters and ParquetValueWriters no longer expose the row column, and ORC's row-not-null precondition is dropped
ORC/Parquet transformPaths removed. Its only reader was the row-data branch
RewriteTablePathUtil.PositionDeleteReaderWriter: the rowSchema overload is gone and the 4-arg writer() is now abstract.

Engines specific

GenericFileWriterFactory: both deprecated constructors and Builder.positionDeleteRowSchema removed, along with the newPositionDeleteWriter override that existed only to honour the row schema. Also removed 9 vestigial configure* overrides which override nothing now that BaseFileWriterFactory is gone (a FormatModelRegistry leftover rather than a position-delete concern, but in the same file).
SparkFileWriterFactory ×3, SparkPositionDeltaWrite ×3, FlinkAppenderFactory ×3: deprecated constructors and row-schema wiring removed; FlinkAppenderFactory.newPosDeleteWriter now builds path+pos writers.
SparkPositionDeletesRewrite ×3: DeleteWriter collapses from two writer factories to one.
RewriteTablePathSparkAction ×3: SparkPositionDeleteReaderWriter implements the 4-arg writer().

Behavior changes worth a close look

Reads of row stay, writes are gone. The position_deletes metadata table still has a row column for backward compatibility with existing v2 delete files; it is null for anything written from now on.
Path rewriting drops the row column. RewriteTablePathUtil previously detected row data in a source delete file and reproduced it; it can no longer do so, so rewritten position delete files contain path+pos only. Same for the rewrite_position_delete_files path in SparkPositionDeletesRewrite, which no longer propagates incoming rows.
Existing callers that set a row schema now fail loudly instead of writing rows. Follows the Testing the effect of the removal of the position delete with row data feature #13870's approach to fail early

`position_deletes` metadata table and tests

The table keeps exposing its row column so that delete files written before 1.12.0 still read back. Nothing can populate it any more, so it reads as null for anything written by this version.

Related test which writes PDWR gets dropped

Verification

gradlew check

.palantir/revapi.yml gains 16 entries across core, data, orc and parquet

Extra Callout

d39351b include change for PositionDeltaWriter.delete(CharSequence, long, T, PartitionSpec, StructLike) still takes a row parameter that is now ignored by BasePositionDeltaWriter. It is not deprecated, so narrowing it belongs in a 1.13.0 deprecation cycle.

AI Disclosure

Model: Claude Opus 5 (1M context)
Platform/Tool: Claude Code
Human Oversight: reviewed
Prompt Summary: split #16449 into smaller self-contained PRs; verify each group compiles and tests green standalone

@pvary if you can help take a look at the change?
CC @szehon-ho for changes relate to position delete metadata table and PDWR in rewrite table path changes

@disabled

Position deletes that carry row data were deprecated in 1.11.0 for removal in 1.12.0, and the spec documents them as unsupported. Following the approach prototyped in apache#13870, the writers reject row data rather than silently dropping it. Writes are rejected; reads are preserved for backward compatibility with delete files produced by earlier versions. Core, ORC, Parquet: - PositionDelete: removed set(CharSequence, long, R) and row(). The row field, size()==3 and the positional get/set are retained so delete files that already carry row data still read back; get(2) is null for anything written from now on. - Avro, ORC and Parquet buildPositionWriter() throw UnsupportedOperationException when a rowSchema is set. rowSchema itself stays on all three DeleteWriteBuilders, since equality deletes require it. - Avro.PositionAndRowDatumWriter is deleted, being unreachable. The PositionDeleteStructWriter in GenericOrcWriters and ParquetValueWriters no longer exposes the row column, and ORC's row-not-null precondition is dropped because it can no longer fire. - ORC and Parquet transformPaths is removed; its only reader was the row-data branch. The path+pos branch always hardcoded Function.identity(), so path handling is unchanged. - BasePositionDeltaWriter and BaseTaskWriter use set(path, pos). - RewriteTablePathUtil.PositionDeleteReaderWriter: the rowSchema overload is removed and the 4-arg writer() is now abstract. Rewritten position delete files no longer reproduce the row column. - PositionDeleteWriter's constructor is narrowed from FileAppender<? extends StructLike> to FileAppender<PositionDelete<T>>, as its 1.11.0 deprecation note scheduled for 1.12.0. Dropping the wildcard removes the unchecked cast; every caller already builds an appender of the exact type, so no call site changes. Erasure is unchanged, so this is a source-only break and revapi reports nothing. - PositionDeltaWriter.delete(CharSequence, long, T, PartitionSpec, StructLike) now accepts and ignores its row argument, so it is deprecated in favour of delete(CharSequence, long, PartitionSpec, StructLike) with removal scheduled for 1.13.0. The four-argument form is currently a default that delegates to the five-argument one; in 1.13.0 that flips, so the four-argument form becomes the method implementations override. BasePositionDeltaWriter is the only implementation and its override is marked accordingly. Data and engines: - GenericFileWriterFactory: both deprecated constructors and Builder.positionDeleteRowSchema are removed, along with the newPositionDeleteWriter override that existed only to honour the row schema. The surviving constructor keeps writerProperties, which Builder.build() sets and which has callers in Flink RowDataTaskWriterFactory and kafka-connect RecordUtils. Also removed 9 vestigial configure* overrides that override nothing now that BaseFileWriterFactory is gone. - SparkFileWriterFactory and SparkPositionDeltaWrite (all versions): deprecated constructors and row-schema wiring removed. - SparkPositionDeletesRewrite: DeleteWriter collapses from two writer factories to one, since incoming rows are no longer propagated. - RewriteTablePathSparkAction: SparkPositionDeleteReaderWriter implements the 4-arg writer(). - FlinkAppenderFactory is removed outright rather than updated, in all three Flink versions. It was deprecated in 1.11.0 in favour of FlinkFileWriterFactory, with class-level javadoc scheduling removal for 1.12.0, and its only references were the class itself and TestFlinkAppenderFactory. Tests: - The position_deletes metadata table keeps its row column so pre-1.12 delete files still read, but nothing can populate it, so tests that write rows and assert on them are dropped: TestPositionDeletesTable testSelect, testPartitionedTable, testArrayColumnFilter, testSchemaEvolutionAdd/Remove and testWriteSchemaEvolutionAdd/Remove. Each already declared the dependency with assumeThat(formatVersion) ... "DVs don't have row info in PositionDeletesTable". - Dropped TestAppenderFactory.testPosDeleteWriterWithRowSchema, TestFileWriterFactory.testPositionDeleteWriterWithRow and TestRewriteTablePathsAction.testPositionDeleteWithRow, plus the @disabled overrides that already anticipated their removal. - The Avro, ORC and Parquet row-data writer tests are replaced by ones asserting the new rejection. - TestWriterMetrics keeps checkRowStatistics unchanged; it already asserts the bounds contain no row fields, which is now the only possible outcome. - SimpleDataUtil.writeEqDeleteFile/writePosDeleteFile's FileAppenderFactory overloads go with FlinkAppenderFactory, which was the only FileAppenderFactory<RowData> implementation, so no Flink test can supply an argument to them. Neither had callers left: TestFlinkManifest, TestIcebergCommitter and TestIcebergFilesCommitter all pass a FileWriterFactory and resolve to the other overload, and TestIcebergFilesCommitter.writePosDeleteFile, the only caller of the position-delete overload, was itself already uncalled. revapi.yml records justifications for the resulting binary breaks.

uros-b · 2026-08-18T06:40:06Z

Nit: flink/{v1.20,v2.0,v2.1}/FlinkAppenderFactory.java is removed entirely, but the PR body describes it as modified ("newPosDeleteWriter now builds path+pos writers"). The deletion is legitimate, the class is @deprecated since 1.11.0, will be removed in favor of FlinkFileWriterFactory, and is unreferenced in main, but its full removal spans a broader migration than PDWR removal alone - please confirm the scope and possibly correct the PR body description accordingly.

pvary · 2026-08-18T07:15:56Z

+   * @deprecated since 1.12.0, will be removed in 1.13.0; the {@code row} is ignored, see {@link
+   *     PositionDeltaWriter#delete(CharSequence, long, Object, PartitionSpec, StructLike)}.


Should this be without the Object?

Suggested change

* @deprecated since 1.12.0, will be removed in 1.13.0; the {@code row} is ignored, see {@link

* PositionDeltaWriter#delete(CharSequence, long, Object, PartitionSpec, StructLike)}.

* @deprecated since 1.12.0, will be removed in 1.13.0; the {@code row} is ignored, see {@link

* PositionDeltaWriter#delete(CharSequence, long, PartitionSpec, StructLike)}.

pvary · 2026-08-18T07:39:52Z

-                .writeProperties(writeProperties)
-                .build();
-        SparkFileWriterFactory writerFactoryWithoutRow =
+        SparkFileWriterFactory writerFactory =


Could we add to the class javadoc, that if there is row-data in the position deletes then it will be removed after the rewrite?

Same for the other Spark versions

dramaticlly · 2026-08-19T00:29:36Z

Convert to draft for now, as current change in rewrite-position-delete and rewrite-table-path direct drop the row for existing position delete with row data, need to think of a better way to handle the writer deletion while not cause trouble for existing workload. Maybe both need to fail early if detected existing PDWR

github-actions Bot added spark parquet core data flink ORC labels Aug 18, 2026

nssalian requested a review from pvary August 18, 2026 03:52

dramaticlly force-pushed the 1.12deprecation-position-delete-row-data branch from d39351b to d48f943 Compare August 18, 2026 05:08

uros-b reviewed Aug 18, 2026

View reviewed changes

pvary reviewed Aug 18, 2026

View reviewed changes

nssalian added this to the Iceberg 1.12.0 milestone Aug 18, 2026

dramaticlly marked this pull request as draft August 19, 2026 00:27

manuzhang mentioned this pull request Aug 19, 2026

Remove deprecated methods for 1.12.0 #16445

Open

Navigation Menu

Uh oh!

Core, Data, Spark, Flink: Remove of position delete files with row data - #17706

Core, Data, Spark, Flink: Remove of position delete files with row data#17706
dramaticlly wants to merge 1 commit into
apache:mainfrom
dramaticlly:1.12deprecation-position-delete-row-data

dramaticlly commented Aug 18, 2026 •

edited

Loading

Uh oh!

uros-b Aug 18, 2026

Uh oh!

pvary Aug 18, 2026

Uh oh!

pvary Aug 18, 2026

Uh oh!

pvary Aug 18, 2026

Uh oh!

dramaticlly commented Aug 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		* @deprecated since 1.12.0, will be removed in 1.13.0; the {@code row} is ignored, see {@link
		* PositionDeltaWriter#delete(CharSequence, long, Object, PartitionSpec, StructLike)}.

Uh oh!

Conversation

dramaticlly commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Core

Engines specific

Behavior changes worth a close look

position_deletes metadata table and tests

Verification

Extra Callout

AI Disclosure

Uh oh!

uros-b Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

pvary Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

pvary Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

pvary Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

dramaticlly commented Aug 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dramaticlly commented Aug 18, 2026 •

edited

Loading

`position_deletes` metadata table and tests