Skip to content

Implement high-performance sorted writing support in IcebergIO #38406

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
atognolag wants to merge 21 commits into
base: master
Choose a base branch
from
Draft

Implement high-performance sorted writing support in IcebergIO #38406

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
1a6c642
Implement high-performance sorted writing support in IcebergIO
atognolag May 7, 2026
094c721
Fix multiple-iteration anti-pattern in WriteGroupedRowsToFiles by ext…
atognolag May 7, 2026
f9dd27d
Refactor row sorting calls to be explicitly conditional on table.sort…
atognolag May 7, 2026
ca4ff74
Simplify sorting logic to keep call DRY and use straightforward 'rows…
atognolag May 7, 2026
df8dc69
Rename variable rows to sortedOrUnsortedRows to explicitly communicat…
atognolag May 7, 2026
9120fda
Fix sorted write PR comments: resolve null-ordering bugs and optimize…
atognolag May 7, 2026
75dcc4f
Fix Checkstyle check MissingDeprecated warning in IcebergRowSorter.java
atognolag May 7, 2026
e4c97bf
Remove unused deprecated encodeSortKey method in IcebergRowSorter.java
atognolag May 7, 2026
ecdaf7f
Finalize Hybrid Partitioning and Sorting Architecture in IcebergIO: e…
atognolag May 8, 2026
890fe76
Set HASH as the default distribution mode in IcebergIO and comprehens…
atognolag May 8, 2026
0cecd01
Add testRangeDistribution integration test case for RANGE distributio…
atognolag May 8, 2026
94fa738
Add Javadoc documentation for withAutosharding method in IcebergIO.java
atognolag May 8, 2026
f24183a
Update RANGE sharding code sample to use ID partitioning and document…
atognolag May 8, 2026
670aab1
Document the multi-dimensional distribution mode decision matrix in I…
atognolag May 8, 2026
acf0e56
Add Operational Impact column to distribution modes Javadoc table in …
atognolag May 8, 2026
e22d4dc
Fix ClassCastException in IcebergUtils string field copying by using …
atognolag May 8, 2026
1ab4fcf
Add comprehensive test scenarios for dynamic BigDecimal and Integer t…
atognolag May 8, 2026
1e19180
Add thorough test scenarios for Double, Boolean, and Null value to St…
atognolag May 8, 2026
4d20b11
Use catalog.buildTable to set SortOrder during dynamic table creation…
atognolag May 8, 2026
fca10f8
Adding a note and a warning
atognolag May 9, 2026
e576dbf
Adding scratch files
atognolag May 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions scratch/iceberg-scale-test/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
plugins {
id 'java'
id 'application'
}

repositories {
mavenLocal()
mavenCentral()
maven {
url "https://repository.apache.org/snapshots/"
}
}

dependencies {
implementation "org.apache.beam:beam-sdks-java-core:2.74.0-SNAPSHOT"
implementation "org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.74.0-SNAPSHOT"
implementation "org.apache.beam:beam-sdks-java-io-iceberg:2.74.0-SNAPSHOT"
implementation "org.apache.beam:beam-sdks-java-extensions-sorter:2.74.0-SNAPSHOT"
implementation "org.apache.beam:beam-runners-google-cloud-dataflow-java:2.74.0-SNAPSHOT"

implementation "org.apache.iceberg:iceberg-core:1.4.3"
implementation "org.apache.iceberg:iceberg-api:1.4.3"
implementation "org.apache.iceberg:iceberg-data:1.4.3"
implementation "org.apache.iceberg:iceberg-gcp:1.4.3"
implementation "com.google.cloud.bigdataoss:gcs-connector:hadoop2-2.2.26"
implementation "org.apache.hadoop:hadoop-client:3.4.2"
implementation "org.apache.hadoop:hadoop-common:3.4.2"

implementation "org.slf4j:slf4j-api:1.7.30"
implementation "org.slf4j:slf4j-jdk14:1.7.30"

annotationProcessor "com.google.auto.value:auto-value:1.9"
compileOnly "com.google.auto.value:auto-value-annotations:1.9"
}

application {
mainClass = 'org.apache.beam.sdk.io.iceberg.test.IcebergBigQueryScaleTest'
}

java {
sourceCompatibility = JavaVersion.VERSION_11
targetCompatibility = JavaVersion.VERSION_11
}
Binary file added scratch/iceberg-scale-test/gradle/wrapper/gradle-wrapper.jar
View file
Open in desktop
Binary file not shown.
7 changes: 7 additions & 0 deletions scratch/iceberg-scale-test/gradle/wrapper/gradle-wrapper.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-8.14.3-bin.zip
networkTimeout=10000
validateDistributionUrl=true
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists
249 changes: 249 additions & 0 deletions scratch/iceberg-scale-test/gradlew
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions scratch/iceberg-scale-test/settings.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
rootProject.name = 'iceberg-scale-test'
Loading
Loading