Skip to content

[core] Support nested-key-null-strategy in FieldNestedUpdateAgg operator#8374

Open
PyRSA wants to merge 4 commits into
apache:masterfrom
PyRSA:feature/nested-update-nested-key-null-strategy
Open

[core] Support nested-key-null-strategy in FieldNestedUpdateAgg operator#8374
PyRSA wants to merge 4 commits into
apache:masterfrom
PyRSA:feature/nested-update-nested-key-null-strategy

Conversation

@PyRSA

@PyRSA PyRSA commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

Purpose

[Feature] Add fields.{fieldName}.nested-key-null-strategy configuration for nested_update function
[Feature] Support nested-key null handling strategies in FieldNestedUpdateAgg operator to control behavior when nested-key does not satisfy primary key semantics

Expected behavior when nested-key contains null values or does not satisfy primary key semantics:

In the following examples, nested-key is defined as k0,k1.

case Input A Input B Current Behavior Expected Behavior
Default behavior (no config) [(0, null, "A", 1)] [(0, 1, "B", 2)] Merge Merge
MERGE strategy [(0, null, "A", 1)] [(0, 1, "B", 2)] Merge Merge
IGNORE strategy [(0, null, "A", 1)] [(0, 1, "B", 2)] Merge ❌ Ignore invalid nested-key row
ERROR strategy [(0, null, "A", 1)] [(0, 1, "B", 2)] Merge ❌ Throw exception

Behavior Definition

  • merge
    Merge rows even if nested-key does not satisfy primary key semantics.
    This is equivalent to not configuring nested-key-null-strategy.

  • ignore
    Ignore rows whose nested-key does not satisfy primary key semantics.

  • error
    Throw an exception when nested-key does not satisfy primary key semantics.


Tests

  • org.apache.paimon.mergetree.compact.aggregate.FieldAggregatorTest#testFieldNestedUpdateAggWithNestedKeyNullStrategyArgumentCheck
  • org.apache.paimon.mergetree.compact.aggregate.FieldAggregatorTest#testFieldNestedUpdateAggWithoutNestedKeyNullStrategy
  • org.apache.paimon.mergetree.compact.aggregate.FieldAggregatorTest#testFieldNestedUpdateAggWhenNestedKeyNullUseMergeStrategy
  • org.apache.paimon.mergetree.compact.aggregate.FieldAggregatorTest#testFieldNestedUpdateAggWhenNestedKeyNullUseIgnoreStrategy
  • org.apache.paimon.mergetree.compact.aggregate.FieldAggregatorTest#testFieldNestedUpdateAggWhenNestedKeyNullUseThrowErrorStrategy
  • org.apache.paimon.mergetree.compact.aggregate.FieldAggregatorTest#testFieldNestedUpdateAggWithCountLimitWhenNestedKeyNullUseMergeStrategy
  • org.apache.paimon.mergetree.compact.aggregate.FieldAggregatorTest#testFieldNestedUpdateAggWithCountLimitWhenNestedKeyNullUseIgnoreStrategy
  • org.apache.paimon.mergetree.compact.aggregate.FieldAggregatorTest#testFieldNestedUpdateAggWithCountLimitWhenNestedKeyNullUseThrowErrorStrategy

API and Format

No Changes.


Documentation

docs/docs/primary-key-table/merge-engine/aggregation.mdx

AuroraVoyage added 4 commits June 28, 2026 15:05
…-null-strategy' into feature/nested-update-nested-key-null-strategy
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant