[Draft RFC] Update guidance on versioning to use parquet-format version by emkornfield · Pull Request #588 · apache/parquet-format

emkornfield · 2026-06-22T06:37:44Z

Concrete proposal for discussion: https://lists.apache.org/thread/8qjyqrbkwj14hrpk8f806bwy2k3m3ds1

steveloughran

minor comments

steveloughran · 2026-06-22T15:04:25Z

@@ -131,62 +138,67 @@ For the purposes of this discussion we classify features into the following buck

 3. Forward incompatible. A file written under a newer version of the format with
   the feature enabled cannot be read under an older version of the format (e.g.


"read under an older version of the format" is a bit confusing here. Presumably it means "by a library/app only written to work with earlier versions"

steveloughran · 2026-06-22T15:09:19Z

+| Field          | Type    | Offset | Description                                                       |
+|----------------|---------|--------|-------------------------------------------------------------------|
+| `metadata_len` | i32 LE  | 0      | Byte length of the Thrift-encoded `FileMetaData` block            |
+| `flags`        | u32 LE  | 4      | Feature flags (see [Feature Flags](#feature-flags))               |


That's a permanent restriction to 32 flags; using a u64 would add minimal cost at this point and avoid complications if 33 flags were ever needed many years in the future

I had the same thought, but I can also understand keeping this simple for now, as I think the intent is to use this bitfield sparingly. Most changes will be communicated with a major version bump, so it's only structural changes like a new footer format that will require a new bit position. I think any new footer we design should itself be extensible without requiring the use of additional bits here. As to clean-ups of the existing thrift structure, perhaps we should roll up the path_in_schema change with other changes that have been suggested before (remove encodings from ColumnMetaData, remove file_offset and file_path from ColumnChunk, I think @alkis had others).

If we are concerned about running out of bits, then we could instead say the footer isn't fixed in size, and use continuation bits to make the bitfield arbitrary in size. Or keep it fixed, and say bit 32 signifies an additional bitfield located in a TBD location in the footer.

I don't feel too strongly. I think even the ones listed only gets us less then 10 bits. And if we end up going with a new footer hopefully that would take 1 bit and hopefully keep this pretty low.

I do think it is a good idea to update the wording that this might not be fixed size (could grow in the future, but that would be detectable via inspecting the bitmap). Another option, given we expect this field to not evolve too quickly past an initial set of changes is to have bit 31, effectively be a version bit, which indicates all structural changes as of a certain date are always true.

One nice thing about 32 bits is it makes the current implementation a little easier since a valid parquet file today must always be at least 16 bytes (2*PAR1, 4 bytes required fields in thirft, 4 bytes metadata length).

steveloughran · 2026-06-22T15:11:13Z

+
+## Integrity Check
+
+The `crc32` field holds a CRC-32 (ISO 3309 / ITU-T V.42 polynomial, the same used for page level CRC values)


crc32s is where integrity checksums should be going, because x86 and ARM parts can do in a single instruction. java has a native method for this now.

I went back and forth on this. I don't feel too strongly but having the same CRC used across the file ultimately seemed better, then having a one off difference.

etseidl

Thanks @emkornfield, I really like this idea.

etseidl · 2026-06-22T15:56:39Z

-   released. It is recommended that changing the default value for a forward
-   incompatible feature flag should be clearly advertised to consumers (e.g. via
+3. Major version upgrades should not be enabled by default
+   until 2 years after the parquet-java implementation for the specification has been


Given the structural and procedural changes here, I wonder if the 2 year suggestion can be softened. Maybe just say these changes should not be enabled as defaults until sufficient time has been given for downstream consumers to adapt.

My main concern here is there we don't have full participation from everybody that has a parquet reader so knowing an exact time to release is still somewhat a guess (especially because some people never update there libraries). I think setting expectations that all readers have two years before the potentially start breaking change become widespread is a reasonable balance.

I'm open to other wording here or other views on a policy that actually enables us to upgrade in a reasonable time-frame but minimizes breakages.

etseidl · 2026-06-22T15:57:51Z

+
+| Bit Index| Name                    | Description                                                                                                 |
+|----------|-------------------------|-------------------------------------------------------------------------------------------------------------|
+| 0        | `ENCRYPTED_FOOTER`      | The `FileMetaData` block is encrypted (equivalent to the `PARE` format).                                    |


Should we add these to parquet.thrift as constants?

Seems reasonable, I'm not clear on the support for constants in thrift, I'll do some research if we can align with this approach.

danielcweeks · 2026-06-22T21:40:23Z

@@ -0,0 +1,89 @@
+# PARX Parquet Format Specification


I think we should separate discussion of PARX and decouple it from parquet format versioning. The two are separate and I'm not sure we need to align on this in order to move forward with the format versioning.

The conversation that spurred this thread and a bunch of the discussion involved reader/writer coordination.

In particular the original doc posed these questions:

Is a deliberately curated capability level — "V3" — the right vehicle for setting shared reader/writer expectations, building on issue #384?

How far can and should we go in aligning the magic number, footer version, and release version?

The PARX format is one means of solving this coordination, which I think satisfies requirements from the community, so I'm not sure it makes sense to decouple it completely?

I think if we're going to design a new magic number it should be done independently and then we use that representation to effect a change like this. I don't think they should be bundled together.

If we are talking specifically about the change to path_in_schema, I agree, we can decouple these once we get alignment on the overall direction. The reason for coupling them in this PR is to illustrate how it would be used in practice.

danielcweeks · 2026-06-22T22:18:24Z

+  /** Path in schema
+   *
+   *  Made optional in parquet-format 3.0. If not written
+   *  PARX magic number must be used (PATH_IN_SCHEMA_OMITTED, bit 1 in the
+   *  feature flag bitmap, must be set).
+   **/
+  3: optional list<string> path_in_schema


If we're changing this now, does that mean that we can't make other changes without releasing 3 first?

It feels like this is something we would change prior to a v3 release not make the change, still continue to release v2 features (e.g. new types) and later do a v3 release.

If we're changing this now, does that mean that we can't make other changes without releasing 3 first?

Yes, once merged the next release off of main would be V3, and I expect given community bandwidth this is the most likely outcome

It feels like this is something we would change prior to a v3 release not make the change, still continue to release v2 features (e.g. new types) and later do a v3 release.

My proposal here is we can backport forward compatible changes. I added the following the proposal:

If there is contributor bandwidth forwards-compatible features may be backported to previous major versions of the specification and released as a new minor version.

I think this gets us out of the trap of trying curate releases, and still allows writers that don't want to upgrade a path to back-porting if there is enough demand.

alamb · 2026-07-10T11:02:25Z

+features may be backported to previous major versions of the specification and released
+as a new minor version.
+
+Every forward incompatible change added to the specification requires


~~Since the parquet-format version is not (yet) stored in the file, readers would not be able to tell if they could read a file~~

Update: I see the rest of this doc

If a feature is forward compatible (will be ignored by old readers) why do we need to encode that in a minor version 🤔

If a feature is forward compatible (will be ignored by old readers) why do we need to encode that in a minor version 🤔

This gets to the grouping features discussion. If we want to use versioning as a way of grouping features, there are forward compatible features that we also want to gate by version (not just have the user opt-in like bloom filters). Recent examples:

ieee754 sort order - As an end-user if the library just suddenly started emitting the new sort order and I was depending on the old one for pruning (in a different system) I'd be a little bit surprised.

The flat-buffers as thrift extension footer (I believe now mostly defunct). Also forward compatible, but as an end-user if I didn't upgrade the write version, why did my file sizes now increase by X%

emkornfield · 2026-07-10T15:21:43Z

+------------------+-----------+----------+--------+
+| metadata_len     | flags     | crc32    | 'PARX' |
+------------------+-----------+----------+--------+
+  offset 0          offset 4    offset 8   offset 12


One thing that was called out is that default Thrift parsers will fail hard if they encounter a new unknown enum for a required field. We should probably add a version number here, so that users can provide a useful error message e.g. something like:

Failed to parse footer. New version XX detected.

but I think the order of operations should still be:

If I support all falgs, try to parse the footer.

If footer parsing fails check version for appropriate error message.

This allows readers with a custom thrift parsers to still potentially read most of the file.

emkornfield added 2 commits June 22, 2026 06:22

wip

0327ca3

wip

974df15

emkornfield changed the title ~~Update guidance on versioning to use parquet-format version~~ [Draft RFC] Update guidance on versioning to use parquet-format version Jun 22, 2026

emkornfield mentioned this pull request Jun 22, 2026

[DRAFT RFC] feat(parquet): add PARX format prototype with CRC32C footer and featu… apache/arrow-rs#10177

Draft

emkornfield marked this pull request as draft June 22, 2026 06:51

steveloughran reviewed Jun 22, 2026

View reviewed changes

etseidl reviewed Jun 22, 2026

View reviewed changes

danielcweeks reviewed Jun 22, 2026

View reviewed changes

emkornfield added 2 commits June 22, 2026 23:21

mention backports

bd585cb

fix typ

07d868f

emkornfield mentioned this pull request Jun 23, 2026

Document Parquet Features by Version apache/parquet-site#186

Merged

remove semver reference

389ba54

alamb reviewed Jul 10, 2026

View reviewed changes

emkornfield commented Jul 10, 2026

View reviewed changes

emkornfield mentioned this pull request Jul 13, 2026

RFC: Versioning only #598

Open

		@@ -131,62 +138,67 @@ For the purposes of this discussion we classify features into the following buck

		3. Forward incompatible. A file written under a newer version of the format with
		the feature enabled cannot be read under an older version of the format (e.g.


		## Integrity Check

		The `crc32` field holds a CRC-32 (ISO 3309 / ITU-T V.42 polynomial, the same used for page level CRC values)

Uh oh!

Conversation

emkornfield commented Jun 22, 2026

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

etseidl left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emkornfield Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emkornfield Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

emkornfield Jun 22, 2026 •

edited

Loading

alamb Jul 10, 2026 •

edited

Loading

emkornfield Jul 10, 2026 •

edited

Loading