[lake/iceberg] Support clean and legacy Iceberg lake table schemas - #4019
Open
fhan688 wants to merge 3 commits into
Open
[lake/iceberg] Support clean and legacy Iceberg lake table schemas#4019fhan688 wants to merge 3 commits into
fhan688 wants to merge 3 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Implements FIP-27 for Iceberg lake tables by introducing “clean” schemas (user columns only) while keeping legacy tables (with __bucket/__offset/__timestamp) fully readable/writable via schema-based layout detection.
Changes:
- Add schema-based legacy detection (
IcebergUtils.isLegacyTable) and thread legacy/clean behavior through writers/readers/planners. - Stop appending Iceberg system columns for newly created tables; preserve legacy layout for compatibility checks and certain evolutions.
- Update partition/sort handling and adjust tests to validate clean-table behavior (unpartitioned/unsorted where applicable).
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| fluss-lake/fluss-lake-iceberg/src/main/java/org/apache/fluss/lake/iceberg/utils/IcebergUtils.java | Adds legacy-vs-clean layout detection helper. |
| fluss-lake/fluss-lake-iceberg/src/main/java/org/apache/fluss/lake/iceberg/IcebergSchemaUtils.java | Introduces clean vs legacy schema builders and renames system column map. |
| fluss-lake/fluss-lake-iceberg/src/main/java/org/apache/fluss/lake/iceberg/IcebergLakeCatalog.java | Makes schema evolution and sort order conditional on legacy layout. |
| fluss-lake/fluss-lake-iceberg/src/main/java/org/apache/fluss/lake/iceberg/utils/IcebergPartitionSpecUtils.java | Skips legacy identity(__bucket) partitioning for clean bucket-unaware tables. |
| fluss-lake/fluss-lake-iceberg/src/main/java/org/apache/fluss/lake/iceberg/utils/IcebergConversions.java | Makes bucket partition key + bucket filter conditional on spec/schema shape. |
| fluss-lake/fluss-lake-iceberg/src/main/java/org/apache/fluss/lake/iceberg/tiering/RecordWriter.java | Threads legacy-layout flag into tiering record conversion. |
| fluss-lake/fluss-lake-iceberg/src/main/java/org/apache/fluss/lake/iceberg/tiering/FlussRecordAsIcebergRecord.java | Emits system columns only for legacy tables; adjusts positional mapping. |
| fluss-lake/fluss-lake-iceberg/src/main/java/org/apache/fluss/lake/iceberg/source/IcebergSplitPlanner.java | Updates bucket/partition extraction for clean vs legacy partition specs. |
| fluss-lake/fluss-lake-iceberg/src/main/java/org/apache/fluss/lake/iceberg/source/IcebergRecordReader.java | Projects/reads offset+timestamp only for legacy tables; emits sentinel otherwise. |
| fluss-lake/fluss-lake-iceberg/src/main/java/org/apache/fluss/lake/iceberg/source/IcebergRecordAsFlussRow.java | Computes business-field count dynamically to handle clean vs legacy/projections. |
| fluss-lake/fluss-lake-iceberg/src/main/java/org/apache/fluss/lake/iceberg/maintenance/IcebergRewriteDataFiles.java | Skips __offset-based sorting when offset column is absent. |
| fluss-lake/fluss-lake-iceberg/src/test/java/org/apache/fluss/lake/iceberg/IcebergLakeCatalogTest.java | Updates expectations for clean schemas, partition specs, and unsorted tables. |
| fluss-lake/fluss-lake-iceberg/src/test/java/org/apache/fluss/lake/iceberg/flink/FlinkCatalogLakeTest.java | Updates schema size expectation to user columns only. |
| fluss-lake/fluss-lake-iceberg/src/test/java/org/apache/fluss/lake/iceberg/testutils/FlinkIcebergTieringTestBase.java | Makes offset assertions conditional; handles clean tables without __offset. |
| fluss-lake/fluss-lake-iceberg/src/test/java/org/apache/fluss/lake/iceberg/tiering/IcebergSchemaEvolutionITCase.java | Updates schema evolution assertions for clean “append-last” behavior. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
|
||
| public void setFlussRecord(LogRecord logRecord) { | ||
| this.logRecord = logRecord; | ||
| this.internalRow = logRecord.getRow(); |
Comment on lines
+72
to
+77
| boolean lastIsLegacyBucket = | ||
| lastSourceField != null && lastSourceField.name().equals(BUCKET_COLUMN_NAME); | ||
| boolean lastIsBucketTransform = lastField.transform().toString().startsWith("bucket["); | ||
| if (lastIsLegacyBucket || lastIsBucketTransform) { | ||
| partitionKey.set(pos, bucket); | ||
| } |
Comment on lines
+161
to
+164
| Types.NestedField lastSourceField = table.schema().findField(lastField.sourceId()); | ||
| boolean lastIsLegacyBucket = | ||
| lastSourceField != null && lastSourceField.name().equals(BUCKET_COLUMN_NAME); | ||
| boolean lastIsBucketTransform = lastField.transform().toString().startsWith("bucket["); |
Comment on lines
+35
to
+44
| /** | ||
| * Returns whether the given Iceberg table is a legacy table (has the three trailing system | ||
| * columns). | ||
| * | ||
| * <p>Detection: if the {@code __timestamp} system column exists in the physical schema, this is | ||
| * a legacy table. Clean tables have no system columns. | ||
| */ | ||
| public static boolean isLegacyTable(Schema icebergSchema) { | ||
| return icebergSchema.findField(TIMESTAMP_COLUMN_NAME) != null; | ||
| } |
Comment on lines
+341
to
+344
| // FIP-27: a clean table stores only user columns; only legacy tables carry the | ||
| // trailing __bucket/__offset/__timestamp columns (offset at idx 3). | ||
| if (actualRecord.struct().field(OFFSET_COLUMN_NAME) != null) { | ||
| assertThat(actualRecord.get(3)).isEqualTo(startingOffset++); |
Comment on lines
+423
to
+433
| Iterable<Record> iterable = | ||
| Parquet.read(table.io().newInputFile(file.location())) | ||
| .project(table.schema()) | ||
| .createReaderFunc( | ||
| fileSchema -> | ||
| GenericParquetReaders | ||
| .buildReader( | ||
| table.schema(), | ||
| fileSchema)) | ||
| .build(); | ||
| iterable.forEach(records::add); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: close #3903
Sub-task of the FIP-27 umbrella (#2411): Remove Mandatory System Columns From Fluss Lake Tables.
Today every Iceberg lake table Fluss creates is forced to carry three mandatory system columns (
__bucket,__offset,__timestamp) as its last physical columns. They pollute the schema users see from Iceberg and other engines. Unlike Paimon,__bucketis additionally woven into the physical layout as anidentity(__bucket)partition field for bucket-unaware tables, and__offsetdrives the table sort order.This PR implements the Iceberg part of FIP-27, mirroring the Paimon part (#3902): newly created Iceberg lake tables use a clean physical schema containing only user-defined columns, while existing legacy tables that still carry the three system columns remain fully readable and writable without any schema migration. Both layouts are supported across create, tiering writers, readers, projections, schema evolution, compaction, and re-enabling tiering.
Brief change log
Layout detection (single source of truth): add
IcebergUtils.isLegacyTable(Schema), defined asschema.findField("__timestamp") != null. Detection is purely schema-based — no new table property or metadata, so existing tables are never migrated.Create:
IcebergSchemaUtils.createIcebergSchemano longer appends the system columns; new tables are clean.SYSTEM_COLUMNSis renamed toLEGACY_SYSTEM_COLUMNS, and a newcreateLegacyIcebergSchemareproduces the legacy layout for compatibility checks. The user-column name-conflict check against system names is kept.Partition spec (Iceberg-specific): in
IcebergPartitionSpecUtils, a bucket-unaware clean table leaves the spec unpartitioned instead of addingidentity(__bucket)(clean tables have no__bucketcolumn). Legacy tables keep theidentity(__bucket)partition; bucket-aware tables keep theirbucket(userCol)transform.Sort order (Iceberg-specific):
IcebergLakeCatalog.createSortOrderreturnsSortOrder.unsorted()for clean tables, which have no__offsetto sort by; legacy tables keepasc(__offset).Schema evolution:
applySchemaChangesinserts a new business column before the first system column only for a legacy table; for a clean table it is appended normally.AddColumnrejects names that collide withLEGACY_SYSTEM_COLUMNSviaInvalidTableException.Compatibility / re-enable tiering:
isIcebergSchemaCompatibledetects the existing layout and, for a legacy table, compares against the legacy schema (createLegacyIcebergSchema) so disabling and re-enabling tiering preserves the physical layout.IcebergPartitionSpecValidatorforks the same way when validating the target spec.Row tiering writer:
FlussRecordAsIcebergRecordemits the three system values only for legacy tables; for clean tables the business-field count equals the full row and no system fields are written. The layout flag is threaded throughRecordWriter.Split planning / bucket extraction (Iceberg-specific):
IcebergSplitPlanner.createBucketEy-spec case (clean bucket-unaware table →bucket = -1) and detects the bucket slot bytransform type (bucket[…]) rather than by source column name, so clean bucket-aware tables — which carry abucket(userCol)transform but no__bucket` column — work correctly.Partition / filter conversion (Iceberg-specific):
IcebergConversions.toPartitionsets the bucket slot only when the last partition field is a legacyidentity(__bucket)or abucket[…]transform;toFilterExpressionadds theequal(__bucket, bucket)predicate only for legacy tables.Compaction:
IcebergRewriteDataFilesnull-guards the__offsetlookup before building t, so clean tables compact without system columns.Reader / projection:
IcebergRecordReaderprojects and reads__offset/__timestamponly for legacy tables. For clean tables it emits a sentinel-1Llog offset / timestamp, consistent with the existingUNKNOWN_OFFSET = -1convention.Row adapter:
IcebergRecordAsFlussRowcomputes the business-field count dynamically fromad of hard-subtracting the three trailing system columns, which also fixes latent miscounts onprojected rows.
Tests
Adapted
IcebergLakeCatalogTest(create/alter assertions now expect user-only schemas, unparIcebergSchemaEvolutionITCase(column-order assertions changed fromisLessThan(indexOf("__bucket"))toisEqualTo(fieldNames.size() - 1)), and theFlinkIcebergoffset assertions guarded on__offset` presence) to the clean layout.Legacy coverage retained:
IcebergTieringTest,IcebergPartitionSpecValidatorTest, andIcebergSplitPlannerTestcontinue to exercise the legacyidentity(__bucket)/asc(__offset)paths;IcebergSplitPlannerTestadditionally covers the empty-spec (clean bucket-unaware) and cleanbucket(col)transform cases.Verified locally on JDK 11: unit tests
IcebergLakeCatalogTest(40),IcebergTieringTest(6lidatorTest(9),IcebergSplitPlannerTest(6) pass; ITsIcebergTieringITCase(1),IcebergSchemaEvolutionITCase(5),IcebergRewriteITCase(3), andFlinkUnionRead*IcebergITCase` union-read cases all pass in isolation.API and Format
No public API change. This changes the physical schema of newly created Iceberg lake tables (clean layout by default) and, for bucket-unaware tables, removes the
identity(__bucket)partition andasc(__offset)sortorder from newly created tables. Existing tables are not migrated and keep their current physic rolling-upgrade requirements are covered by the umbrella #2411 and documented in #3905:
Documentation
Feature behavior (clean vs. legacy layouts, detection, and the rolling-upgrade/compatibility mately under #3905. No standalone doc change in this PR.