Site: Expand Variant intro blog with Iceberg table-level details - #17705
Open
nssalian wants to merge 1 commit into
Open
Site: Expand Variant intro blog with Iceberg table-level details#17705nssalian wants to merge 1 commit into
nssalian wants to merge 1 commit into
Conversation
uros-b
reviewed
Aug 18, 2026
|
|
||
| ### Statistics and data skipping | ||
|
|
||
| Because Variant is a column in the Iceberg schema, the table's manifests carry statistics for it, and that is what lets Iceberg skip files during planning. Iceberg records value and null counts for a Variant column. When a field is shredded into its own typed column, Iceberg also records lower and upper bounds for it, stored as a Variant object whose keys are normalized JSON paths to each field. An unshredded `value` blob is opaque, so it contributes counts but no bounds. |
Member
There was a problem hiding this comment.
The spec contradicts the final sentence. The "Bounds for Variant" section of format/spec.md defines a general mechanism (valid for shredded and unshredded Variant data), where the Variant column's manifest lower_bounds/upper_bounds entries are stored as Variant objects whose keys are normalized JSON path expressions (e.g. $['event_type']) and whose values are per-field min/max primitives. The spec explicitly states: "Including bounds for any field is optional", it does not restrict bounds to files written with shredding.
Suggested fix: Change the paragraph to reflect the practical picture, e.g.
For an unshredded Variant column, computing per-field bounds requires reading the raw value bytes, so writers typically record only counts. When a field is shredded into its own typed column, its Parquet statistics are available directly, and Iceberg can record lower and upper bounds for that field path in the manifest, stored as a Variant object whose keys are normalized JSON paths.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Discussed with @kevinjqliu offline about enhancing the blog for variant to have more Iceberg internal information in addition to the current content, so I: