Introduces a new LogicalType: FILE by brkyvz · Pull Request #585 · apache/parquet-format

brkyvz · 2026-06-09T22:04:45Z

Rationale for this change

Introduces a new type called File as a typed FileReference. The design document is here.

The motivation is as follows:

Unstructured data ingestion is getting extremely popular with the advances in Generative AI. 
Today, our only means of dealing with unstructured data is to store it as a binary blob inside Parquet, 
or point to files that exist in some object store with a string. These solutions fail to address these use 
cases, because of scalability, usability, and governance issues.

We would like to introduce a new logical type annotation in Parquet called “File” for storing a struct that 
contains a path reference to a file with additional metadata. This reference may be to a file that exists 
(or expected to exist) in storage at a given path. We’d like to define the minimum required list of fields 
that would allow a client to correctly read the referenced data. Any additional metadata can be optionally 
stored by engines and table formats as necessary adjacent to this type.

What changes are included in this PR?

Introduces the specification for FileType.

Do these changes have PoC implementations?

Yes:

emkornfield

Looks reasonable to me a few minor comments for clarification.

etseidl

Just a few questions I have after reviewing the Rust implementation.

alkis · 2026-07-06T07:28:27Z

+The referenced bytes are compressed with the same `CompressionCodec` as the one
+specified for the `inline` column.


Let's make this clearer.

Suggested change

The referenced bytes are compressed with the same `CompressionCodec` as the one

specified for the `inline` column.

The bytes referenced by a self-reference are compressed with the same `CompressionCodec` as the one specified for the `inline` column.

Should we point out CompressionCodec can differ per page?

@alkis thoughts? Is there a reason that the data needs to be compressed additionally? Most formats may be optimal anyway.

The reason is because we are making the assumption that the type of data stored in the file column is homogeneous. I argue that's a good assumption. If this column contains text, there is little reason to assume it will be a string when small and an image when large. Ergo if the writer chose to compress the column it should compress the self-referenced packed blobs too.

I wanted to make all packed references compressed as well but there was some pushback for that with the argument that external may be referenced by different parquet files and it may choose its own compression scheme (or none at all). I can see arguments both ways - it is a tradeoff.

Making all packed references inherit the CompressionCodec makes the spec more consistent.
Making only self-references inherit the CompressionCodec is friendlier to shared external references.

We pick one of the two, it is a tradeoff.

Should we point out CompressionCodec can differ per page?

etseidl

Thanks @brkyvz, this looks good to me now.

sfc-gh-sgrafberger

Thank you @brkyvz! Looks good to me.

…a-io#585) Align with the latest apache/parquet-format#585: - size must be set whenever offset is set; a self-reference (no path) must set offset, and therefore size. Drop the now-invalid [offset, EOF) and [0, size) self-reference modes and the external [offset, EOF) mode from the resolution table; add explicit invalid rows. - Define "set" (present, non-null, non-empty for strings) and allow sparse group definitions (a group need only define the fields it uses); add an inline-only example group. - Fields matched case-sensitively by name; field IDs "if they exist". - Readers should ignore unknown checksum algorithms. Follows the consistent prose intent of PR delta-io#585; note its resolution table still lists an [offset, EOF) row that contradicts its own validation section (offset requires size) -- to be raised on the Parquet PR. Co-authored-by: Isaac

brkyvz · 2026-07-10T15:53:47Z

Addressed your comments, @rok @pitrou please take a look. There's only one unaddressed comment on the CompressionCodec, which I will address after talking to @alkis

rdblue · 2026-07-13T20:50:02Z

+* A self-reference (`path` not set) must set `offset`. A value with neither `path` nor
+  `offset` set (and not `inline`) does not resolve and is invalid.
+* `size` must be set whenever `offset` is set. A value that sets `offset` without `size`
+  is invalid. Because a self-reference must set `offset`, it must also set `size`.


How should implementations behave if offset is set but size is not? Are they required to fail?

Ideally fail at write time, but is there a precedent for enforcing that? From your past comments it seems like we don't necessarily enforce contents of columns. Invalid files can be treated as null files in my opinion if we can't enforce it

A "must" requirement is sufficient for writers. I think that requirement implies a failure at write time because otherwise the requirements are violated.

I'm more concerned about read time, which is why I'm asking. If this is invalid, what is the expected reader behavior? In the offset=null case, size is understood to be the length of the file. That doesn't seem right, but my guess is that readers will produce bytes up to the end of the file unless you specifically state that this should fail.

rdblue

I left a few comments for clarification, but I think that this is ready either way.

wgtmac · 2026-07-14T16:13:40Z

+the current file, so a file containing self-references is renamed or relocated as a
+single unit.
+
+The bytes referenced by a self-reference are compressed with the same `CompressionCodec`


The does not look right. Same column can have different codecs across different row groups. V2 pages can even decide whether to compress individually. Even if all row groups of the same column use the same codec, how can we tell whether self-referenced data is compressed or not?

reworded to

The bytes referenced by a self-reference use the `CompressionCodec` defined by the `inline` column chunk's `ColumnMetadata`.

is that better @wgtmac ?

I may have missed the relevant discussion on this so please correct me if I was wrong.

CompressionCodec defined by a ColumnMetadata is to compress all data pages of that column chunk in that specific row group. The description here means that we enforce all inline blocks referenced by this FILE column to always use the same CompressionCodec in the same row group? Is this too restrictive? Why can't we choose them to be uncompressed (like bloom filter) or compressed by a different scheme?

For other non-self-referenced cases, we can deduce compression codec based on content_type field, right? Shouldn't we do the same thing?

For other non-self-referenced cases, we can deduce compression codec based on content_type field, right? Shouldn't we do the same thing?

The intent of the field is to capture the content type of the data for filters like "find all PNG images". I don't expect this to describe compression because that wouldn't be very useful (all application/zip, for example).

I think the content_type field would behave like the HTTP Content-Type header that is the original media type before compression. HTTP uses a separate Content-Encoding header for compression.

This leaves the question of how to determine the equivalent of Content-Encoding for external references. We should probably spend some time considering that more.

The description here means that we enforce all inline blocks referenced by this FILE column to always use the same CompressionCodec in the same row group? Is this too restrictive? Why can't we choose them to be uncompressed (like bloom filter) or compressed by a different scheme?

Yes, I think that reading is accurate. Looks like this is a way to determine compression, but only for blobs that are located within the Parquet file itself as a self-reference.

I don't think that this is too restrictive since generic compression is typically uniform for files. I don't expect people to customize the compression used for these. We want compression to be possible (mandating uncompressed doesn't seem like a good choice), but we don't need to track it per value.

wgtmac · 2026-07-14T16:16:56Z

+```
+
+Because every field is optional, a group need only define the fields it uses. A group
+whose values are always stored inline may define just `inline`:


Why defined content_type below but here says just inline? Seems inconsistent.

I don't see this is addressed. At least inline_file struct below still defines content_type.

brkyvz · 2026-07-17T21:47:49Z

@rdblue @wgtmac @mapleFU @RussellSpitzer Thank you for the feedback! Addressed your comments. Hopefully things are clearer now! Let me know if they aren't

rok · 2026-07-17T23:22:04Z

+| Algorithm | Encoding      | Notes                                                    |
+|-----------|---------------|----------------------------------------------------------|
+| `ETAG`    | opaque        | the object-store eTag, not recomputable                  |
+| `MD5`     | lowercase hex | as defined in RFC 6151 represented as 32 hex characters  |


I would propose we reference RFC 1321 (The MD5 Message-Digest Algorithm) over RFC 6151 (Updated Security Considerations for the MD5 Message-Digest and the HMAC-MD5 Algorithms).

Suggested change

| `MD5` | lowercase hex | as defined in RFC 6151 represented as 32 hex characters |

| `MD5` | lowercase hex | as defined in RFC 1321 represented as 32 hex characters |

rok · 2026-07-17T23:39:40Z

+| `CRC32`   | lowercase hex | as defined in RFC 3385, represented as 8 hex characters  |
+| `CRC32C`  | lowercase hex | as defined in RFC 9260, represented as 8 hex characters  |


rok · 2026-07-17T23:50:10Z

+  `offset` set (and not `inline`) does not resolve and is invalid.
+* `size` must be set whenever `offset` is set. A value that sets `offset` without `size`
+  is invalid. Because a self-reference must set `offset`, it must also set `size`.
+* If `inline` is set, it supplies the bytes readers; producers may treat `inline` and the


Suggested change

* If `inline` is set, it supplies the bytes readers; producers may treat `inline` and the

* If `inline` is set, it supplies the bytes; producers may treat `inline` and the

rok · 2026-07-18T09:14:22Z

+| set      | –      | –        | –      | the inline bytes                                      |
+| –        | set    | –        | –      | whole external file at `path`                         |
+| –        | set    | set      | -      | invalid                                               |
+| –        | set    | –        | set    | external `path`, `[0, size)`                          |
+| –        | set    | set      | set    | external `path`, `[offset, offset + size)`            |
+| –        | -      | set      | -      | invalid                                               |
+| –        | -      | -        | set    | invalid                                               |
+| –        | –      | set      | set    | this file, `[offset, offset + size)` (self-reference) |
+| –        | –      | –        | –      | nothing — invalid                                     |


nit: this table uses both dash (-) and em dash (–) to indicate null input. Let's just use dash?.

Add FILE type definitions

7d966f0

emkornfield reviewed Jun 10, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

emkornfield reviewed Jun 10, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

emkornfield approved these changes Jun 10, 2026

View reviewed changes

danielcweeks reviewed Jun 10, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

danielcweeks reviewed Jun 10, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

Address comments

b6ae0de

RussellSpitzer approved these changes Jun 10, 2026

View reviewed changes

danielcweeks approved these changes Jun 10, 2026

View reviewed changes

etseidl reviewed Jun 10, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

Comment thread LogicalTypes.md Outdated

wgtmac reviewed Jun 11, 2026

View reviewed changes

Comment thread src/main/thrift/parquet.thrift Outdated

Comment thread LogicalTypes.md Outdated

Comment thread LogicalTypes.md Outdated

Comment thread LogicalTypes.md Outdated

Comment thread LogicalTypes.md Outdated

divjotarora reviewed Jun 11, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

rdblue reviewed Jun 11, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

rdblue reviewed Jun 11, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

rdblue reviewed Jun 11, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

rdblue reviewed Jun 11, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

rdblue reviewed Jun 11, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

sfc-gh-sgrafberger reviewed Jun 12, 2026

View reviewed changes

Comment thread LogicalTypes.md

alkis reviewed Jun 17, 2026

View reviewed changes

Comment thread src/main/thrift/parquet.thrift Outdated

CTTY reviewed Jun 17, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

pitrou reviewed Jun 30, 2026

View reviewed changes

Comment thread src/main/thrift/parquet.thrift Outdated

Comment thread src/main/thrift/parquet.thrift Outdated

Comment thread src/main/thrift/parquet.thrift Outdated

Comment thread src/main/thrift/parquet.thrift Outdated

Comment thread LogicalTypes.md Outdated

sfc-gh-sgrafberger reviewed Jul 2, 2026

View reviewed changes

Comment thread src/main/thrift/parquet.thrift

brkyvz added 2 commits July 2, 2026 16:30

update spec based on Alkis' proposal

0f05340

minor changes

bcbabcc

etseidl reviewed Jul 2, 2026

View reviewed changes

alkis reviewed Jul 6, 2026

View reviewed changes

This was referenced Jul 6, 2026

[PROTOCOL RFC] File Data Type delta-io/delta#7147

Open

[Protocol RFC] File data type delta-io/delta#7148

Open

brkyvz added 2 commits July 7, 2026 22:05

address comments

117b3d1

save changes

4ff444e

emkornfield reviewed Jul 8, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

emkornfield reviewed Jul 8, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

emkornfield reviewed Jul 8, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

alkis reviewed Jul 8, 2026

View reviewed changes

Comment thread src/main/thrift/parquet.thrift Outdated

Comment thread LogicalTypes.md Outdated

Comment thread LogicalTypes.md Outdated

Comment thread LogicalTypes.md

Comment thread LogicalTypes.md Outdated

Comment thread LogicalTypes.md Outdated

etseidl approved these changes Jul 8, 2026

View reviewed changes

sfc-gh-sgrafberger approved these changes Jul 8, 2026

View reviewed changes

brkyvz added 2 commits July 8, 2026 18:03

address comments

a60d312

address final comments

f03610e

rok reviewed Jul 9, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

rok reviewed Jul 9, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

address comments, update path spec to URI

8a2ae51

danielcweeks approved these changes Jul 13, 2026

View reviewed changes

rdblue reviewed Jul 13, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

rdblue reviewed Jul 13, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

rdblue reviewed Jul 13, 2026

View reviewed changes

Comment thread LogicalTypes.md

rdblue reviewed Jul 13, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

rdblue reviewed Jul 13, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

rdblue reviewed Jul 13, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

rdblue reviewed Jul 13, 2026

View reviewed changes

Comment thread LogicalTypes.md

rdblue approved these changes Jul 13, 2026

View reviewed changes

mapleFU reviewed Jul 14, 2026

View reviewed changes

Comment thread LogicalTypes.md

wgtmac reviewed Jul 14, 2026

View reviewed changes

RussellSpitzer reviewed Jul 14, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

address comments

90147dc

rok suggested changes Jul 17, 2026

View reviewed changes

rok reviewed Jul 18, 2026

View reviewed changes

		The referenced bytes are compressed with the same `CompressionCodec` as the one
		specified for the `inline` column.

	The referenced bytes are compressed with the same `CompressionCodec` as the one
	specified for the `inline` column.
	The bytes referenced by a self-reference are compressed with the same `CompressionCodec` as the one specified for the `inline` column.

	\| `MD5` \| lowercase hex \| as defined in RFC 6151 represented as 32 hex characters \|
	\| `MD5` \| lowercase hex \| as defined in RFC 1321 represented as 32 hex characters \|

		\| `CRC32` \| lowercase hex \| as defined in RFC 3385, represented as 8 hex characters \|
		\| `CRC32C` \| lowercase hex \| as defined in RFC 9260, represented as 8 hex characters \|

	* If `inline` is set, it supplies the bytes readers; producers may treat `inline` and the
	* If `inline` is set, it supplies the bytes; producers may treat `inline` and the

Uh oh!

Conversation

brkyvz commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Do these changes have PoC implementations?

Uh oh!

Uh oh!

Uh oh!

emkornfield left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

etseidl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

etseidl left a comment

Choose a reason for hiding this comment

Uh oh!

sfc-gh-sgrafberger left a comment

Choose a reason for hiding this comment

Uh oh!

brkyvz commented Jun 9, 2026 •

edited

Loading

wgtmac Jul 14, 2026 •

edited

Loading