Skip to content

fix: check mayContainNulls in ParquetDictionaryRowGroupFilter.notStartsWith - #17682

Open
waterWang wants to merge 2 commits into
apache:mainfrom
waterWang:fix-parquet-notstartsWith-nulls
Open

fix: check mayContainNulls in ParquetDictionaryRowGroupFilter.notStartsWith#17682
waterWang wants to merge 2 commits into
apache:mainfrom
waterWang:fix-parquet-notstartsWith-nulls

Conversation

@waterWang

Copy link
Copy Markdown

Fixes #17655

Problem

ParquetDictionaryRowGroupFilter.notStartsWith can skip row groups that contain null values, silently dropping rows from scan results.

The filter decides whether a row group can be skipped by inspecting only the column's dictionary. A Parquet dictionary contains non-null values only, so when every dictionary entry starts with the prefix the filter returns ROWS_CANNOT_MATCH — even when the column also contains nulls.

In Iceberg a null value matches notStartsWith: Evaluator implements it as !startsWith(...), and startsWith evaluates to false for null. A row group holding nulls therefore does contain matching rows and must not be skipped.

Fix

Add a mayContainNulls check to notStartsWith, consistent with how notEq, notIn, and notNaN already handle null-containing row groups.

Changes

  1. ParquetDictionaryRowGroupFilter.java: Added if (mayContainNulls.get(id)) return ROWS_MIGHT_MATCH; in notStartsWith, matching the pattern used by notNaN.
  2. TestDictionaryRowGroupFilter.java: Updated the assertion for notStartsWith("some_nulls", "some") from isFalse() to isTrue() — this row group contains nulls, so it must be read.

@uros-b uros-b left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is already in progress #17656.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

notStartsWith skips Parquet row groups that contain null values

2 participants