Skip to content

fix: handle empty struct in CompareSchemasVisitor.struct() - #17683

Open
waterWang wants to merge 3 commits into
apache:mainfrom
waterWang:fix-compareschemasvisitor-empty-struct
Open

fix: handle empty struct in CompareSchemasVisitor.struct()#17683
waterWang wants to merge 3 commits into
apache:mainfrom
waterWang:fix-compareschemasvisitor-empty-struct

Conversation

@waterWang

Copy link
Copy Markdown

Fixes #17650

Problem

CompareSchemasVisitor.struct() computes its result as:

Result result = fields.stream().reduce(Result::merge).orElse(Result.SCHEMA_UPDATE_NEEDED);

For a struct with zero fields, the reduce is empty, so orElse(SCHEMA_UPDATE_NEEDED) always fires. Consequently any schema containing an empty struct (e.g. derived from an Avro union whose branch is a zero-field marker record) can never compare SAME with any table schema — including a table created from that exact schema.

In the Dynamic Sink this is fatal: TableUpdater.findOrCreateSchema sees SCHEMA_UPDATE_NEEDED, applies EvolveSchemaVisitor (a no-op here), commits, re-compares — still SCHEMA_UPDATE_NEEDED — and TableMetadataCache.schema() caches and returns NOT_FOUND, whose resolvedTableSchema() is null.

Fix

Change orElse(Result.SCHEMA_UPDATE_NEEDED) to orElse(Result.SAME) — when both structs have zero fields, they are semantically identical.

Changes

  • flink/v1.20/flink/src/main/java/.../CompareSchemasVisitor.java
  • flink/v2.0/flink/src/main/java/.../CompareSchemasVisitor.java
  • flink/v2.1/flink/src/main/java/.../CompareSchemasVisitor.java

All three versions had the same bug (identical content).

@github-actions github-actions Bot added the flink label Aug 16, 2026

@ebyhr ebyhr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a regression test, or do you think it's overkill?

@uros-b uros-b left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 with @ebyhr for adding regression testing. Issue #17650 supplies a ready repro, right @waterWang?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flink: Dynamic Sink — CompareSchemasVisitor never returns SAME for schemas containing empty structs, causing NPE via TableMetadataCache.NOT_FOUND

3 participants