Skip to content

Native Parquet writer derives schema nullability and field IDs from Arrow rather than Catalyst #5305

Description

@andygrove

Describe the bug

ParquetWriterExec builds the output Parquet schema by renaming the fields of the incoming Arrow schema:

let fields: Vec<_> = input_schema.fields().iter().enumerate()
    .map(|(i, field)| Arc::new(field.as_ref().clone().with_name(&column_names[i])))
    .collect();

So two pieces of schema information come from whatever the Comet scan/serde produced, not from the Catalyst schema Spark intends to write:

  1. Nullability. Spark writes each column required or optional according to the Catalyst StructField.nullable of the target schema. Comet inherits the Arrow field's nullability, which can differ — a column Spark would mark required may come out optional, or vice versa. That changes the file's schema as seen by every reader, and a spuriously required column is the dangerous direction.

  2. Field IDs. spark.sql.parquet.fieldId.write.enabled makes Spark emit Parquet field IDs from the Catalyst metadata. Comet emits none, so ParquetFieldIdIOSuite-style round trips and any consumer keyed on field ID (Iceberg, Delta) lose them silently.

Expected behavior

Take nullability and field IDs from the Catalyst schema. WriteJobDescription.dataColumns is already plumbed through to CometWriteFilesExec for column names (#5293), so the attribute list with its nullable flag and metadata is available at the same point — it needs to reach the native side as a serialized schema rather than just names.

Additional context

Found while reviewing #5293. Same family as #3425 (INT96) and #3427 (footer metadata keys): the written file is readable but is not byte-comparable with what Spark writes, which matters most for files other engines read — see #4658 and the plan to write Iceberg data files natively.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions