Skip to content

Site: Expand Variant intro blog with Iceberg table-level details - #17705

Open
nssalian wants to merge 1 commit into
apache:mainfrom
nssalian:blog-enhancement
Open

Site: Expand Variant intro blog with Iceberg table-level details#17705
nssalian wants to merge 1 commit into
apache:mainfrom
nssalian:blog-enhancement

Conversation

@nssalian

Copy link
Copy Markdown
Collaborator

Changes

Discussed with @kevinjqliu offline about enhancing the blog for variant to have more Iceberg internal information in addition to the current content, so I:

  • Covered schema, cross-file layout, snapshots, file formats, and statistics-based data skipping
  • Did some cleanup to prepare for the next blog (shredding)

@github-actions github-actions Bot added the docs label Aug 18, 2026
@nssalian
nssalian requested a review from kevinjqliu August 18, 2026 01:03

### Statistics and data skipping

Because Variant is a column in the Iceberg schema, the table's manifests carry statistics for it, and that is what lets Iceberg skip files during planning. Iceberg records value and null counts for a Variant column. When a field is shredded into its own typed column, Iceberg also records lower and upper bounds for it, stored as a Variant object whose keys are normalized JSON paths to each field. An unshredded `value` blob is opaque, so it contributes counts but no bounds.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The spec contradicts the final sentence. The "Bounds for Variant" section of format/spec.md defines a general mechanism (valid for shredded and unshredded Variant data), where the Variant column's manifest lower_bounds/upper_bounds entries are stored as Variant objects whose keys are normalized JSON path expressions (e.g. $['event_type']) and whose values are per-field min/max primitives. The spec explicitly states: "Including bounds for any field is optional", it does not restrict bounds to files written with shredding.

Suggested fix: Change the paragraph to reflect the practical picture, e.g.

For an unshredded Variant column, computing per-field bounds requires reading the raw value bytes, so writers typically record only counts. When a field is shredded into its own typed column, its Parquet statistics are available directly, and Iceberg can record lower and upper bounds for that field path in the manifest, stored as a Variant object whose keys are normalized JSON paths.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants