Skip to content

fix: use getWithoutDefault to avoid Connect schema default for null values - #17684

Open
waterWang wants to merge 4 commits into
apache:mainfrom
waterWang:fix-kafka-connect-null-default
Open

fix: use getWithoutDefault to avoid Connect schema default for null values#17684
waterWang wants to merge 4 commits into
apache:mainfrom
waterWang:fix-kafka-connect-null-default

Conversation

@waterWang

Copy link
Copy Markdown

Fixes #17652

Problem

Kafka Connect's Struct.get(Field) returns the field's schema defaultValue when the stored value is null, without checking isOptional() (longstanding Connect behavior, see KAFKA-8713). The Iceberg sink and its bundled SMTs read field values with get() in several places, so an explicit NULL in the source ends up written to the Iceberg table as the column's default value.

The most common trigger is Debezium CDC: for any column that is nullable with a non-NULL default (e.g. MySQL VARCHAR(255) NULL DEFAULT ''), every explicitly NULL value is replaced with the default.

Fix

Replace Struct.get(Field) with Struct.getWithoutDefault(String) at all affected call sites. getWithoutDefault returns the actual stored value without applying the schema default, so null values are preserved as null.

Changes

  • DebeziumTransform.java — copy loop in applyWithSchema
  • KafkaMetadataTransform.java — copy loop in applyWithSchema
  • CopyValue.java — copy loop and single-field copy in applyWithSchema
  • RecordConverter.java — field reads in structToIceberg, collectFieldNames, and structToVariant

@uros-b uros-b left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#17653 seems to be a strict superset of these changes, and it seems that it's well in progress - cc @moggaa

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Kafka Connect: explicit NULL values are silently replaced by Connect schema default values

2 participants