Mark shredded_variant cases 41, 131, 132, 138 as INVALID (missing 'value' column)#117
Conversation
…lue' column) These cases omit the required 'value' column from a variant group (cases 41, 131, 138 at the top level; case 132 in the shredded object field groups) but were labeled as valid cases. Per the discussion in apache/parquet-format#591, the spec requires the 'value' field to always be present, so these files are not spec-compliant. Relabel them following the existing convention used by cases 43, 84 and 125: '-INVALID' filenames plus a 'notes' entry stating that implementations can choose to error or read the shredded value. The expected variant outputs are kept for implementations that choose to read them. Fixes apache#116 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Does this break existing CI of dependent projects if merged? |
|
No. I checked the implementations I could find that use this corpus, and they either consume it as a git submodule (arrow-rs, arrow-go, arrow-dotnet, arrow-cpp, datafusion) or pin it to an explicit revision. iceberg-go pins a And when they bump, they are fine. The one project I found that this does affect is hardwood. It shallow-clones parquet-testing master at HEAD (unpinned), enumerates I cannot claim to have found every consumer. parquet-testing has no dependency graph, so this is not exhaustively enumerable. If any implementer pins to master, please shout. |
|
I plan to merge this PR in the next day or two unless anyone would like more time to offer comments |
rdblue
left a comment
rdblue
left a comment
There was a problem hiding this comment.
Seems fine to me. I think implementations should choose to read, but that is for resilience, not adherence to the spec.
|
Thank you everyone for the help |
apache/parquet-testing#117 renamed the shredded_variant fixtures whose variant group omits the spec-required `value` column to carry an `-INVALID` suffix. Our skip-lists keyed on exact pre-rename filenames, so a directory walk still found the renamed files while the skip no longer matched them: once the corpus is re-cloned at HEAD, cases 41/131/138 start being exercised and turn CI red (Hardwood fail-fasts on the malformed group; parquet-java NPEs), and case 132 begins to be compared. Skip these deliberately by the stable `-INVALID` suffix upstream keeps in lockstep with cases.json, rather than by name, so future relabels don't reopen this gap. The suffix now covers case 132 as well: reader behaviour on spec-invalid fixtures is implementation-defined, so pinning it to a byte- or oracle-level match asserts more than the spec guarantees. The one remaining name-keyed group is the fixtures cases.json marks as error cases (an `error_message` and no variant payload). cases.json exposes nothing the `-INVALID` suffix can key on for those, so they stay explicit — but as a single SHREDDED_VARIANT_ERROR_CASES constant shared by both the comparison and reassembly tests rather than duplicated in each.
apache/parquet-testing#117 renamed the shredded_variant fixtures whose variant group omits the spec-required `value` column to carry an `-INVALID` suffix. Our skip-lists keyed on exact pre-rename filenames, so a directory walk still found the renamed files while the skip no longer matched them: once the corpus is re-cloned at HEAD, cases 41/131/138 start being exercised and turn CI red (Hardwood fail-fasts on the malformed group; parquet-java NPEs), and case 132 begins to be compared. Skip these deliberately by the stable `-INVALID` suffix upstream keeps in lockstep with cases.json, rather than by name, so future relabels don't reopen this gap. The suffix now covers case 132 as well: reader behaviour on spec-invalid fixtures is implementation-defined, so pinning it to a byte- or oracle-level match asserts more than the spec guarantees. The one remaining name-keyed group is the fixtures cases.json marks as error cases (an `error_message` and no variant payload). cases.json exposes nothing the `-INVALID` suffix can key on for those, so they stay explicit — but as a single SHREDDED_VARIANT_ERROR_CASES constant shared by both the comparison and reassembly tests rather than duplicated in each.
apache/parquet-testing#117 renamed the shredded_variant fixtures whose variant group omits the spec-required `value` column to carry an `-INVALID` suffix. Our skip-lists keyed on exact pre-rename filenames, so a directory walk still found the renamed files while the skip no longer matched them: once the corpus is re-cloned at HEAD, cases 41/131/138 start being exercised and turn CI red (Hardwood fail-fasts on the malformed group; parquet-java NPEs), and case 132 begins to be compared. Skip these deliberately by the stable `-INVALID` suffix upstream keeps in lockstep with cases.json, rather than by name, so future relabels don't reopen this gap. The suffix now covers case 132 as well: reader behaviour on spec-invalid fixtures is implementation-defined, so pinning it to a byte- or oracle-level match asserts more than the spec guarantees. The one remaining name-keyed group is the fixtures cases.json marks as error cases (an `error_message` and no variant payload). cases.json exposes nothing the `-INVALID` suffix can key on for those, so they stay explicit — but as a single SHREDDED_VARIANT_ERROR_CASES constant shared by both the comparison and reassembly tests rather than duplicated in each.
4 participants
Rationale
These four cases omit the required
valuecolumn from a variant group but are labeled as valid cases with expected outputs. Per the discussion in apache/parquet-format#591, the spec's requirement that the group contain avaluefield is intentional and will not be relaxed, so these files are not spec-compliant. The mislabeling has already propagated into implementations that run this corpus in CI (see apache/arrow-rs#10306).testArrayMissingValueColumnvalue(typed_value is a LIST)testMissingValueColumnvalue(typed_value is INT32)testShreddedObjectMissingFieldValueColumnvalueinside the shredded object field groupstestShreddedObjectMissingValueColumnvalue(typed_value is an object)Changes
Follows the existing convention used by cases 43, 84 and 125:
case-NNN.parquet/case-NNN_row-0.variant.bintocase-NNN-INVALID.parquet/case-NNN-INVALID_row-0.variant.binnotesentry incases.jsonstating the file is not valid according to the spec (with the reason) and that implementations can choose to error, or read the shredded valueNo file contents were changed, only renames and
cases.jsonmetadata.🤖 Generated with Claude Code