Core: Add strict content stats evaluator - #17701
Draft
nastra wants to merge 2 commits into
Draft
Conversation
The strict evaluator assumed that a field has no nulls when a null count is missing for the field, and that a field has no NaNs when a NaN count is unavailable. Both assumptions allow reporting that all rows in a file match a comparison predicate when they may not, because neither null nor NaN matches a comparison predicate. Treat unknown counts as possible nulls and NaNs instead. Optionality is used to determine whether a field can contain nulls, and NaNs are only considered for floating point fields, where NaN counts are tracked.
nastra
marked this pull request as draft
August 17, 2026 18:51
nastra
force-pushed
the
5-strict-stats-evaluator
branch
from
August 17, 2026 19:04
eba78ad to
0e9d9c4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This currently depends on #17648.
It adds the strict content stats evaluator (similar to #17413) and updates the strict metrics evaluator tests to make it possible to run them against the strict metrics & stats evaluators.