Describe the bug
ParquetWriterExec builds the output Parquet schema by renaming the fields of the incoming Arrow schema:
let fields: Vec<_> = input_schema.fields().iter().enumerate()
.map(|(i, field)| Arc::new(field.as_ref().clone().with_name(&column_names[i])))
.collect();
So two pieces of schema information come from whatever the Comet scan/serde produced, not from the Catalyst schema Spark intends to write:
-
Nullability. Spark writes each column required or optional according to the Catalyst StructField.nullable of the target schema. Comet inherits the Arrow field's nullability, which can differ — a column Spark would mark required may come out optional, or vice versa. That changes the file's schema as seen by every reader, and a spuriously required column is the dangerous direction.
-
Field IDs. spark.sql.parquet.fieldId.write.enabled makes Spark emit Parquet field IDs from the Catalyst metadata. Comet emits none, so ParquetFieldIdIOSuite-style round trips and any consumer keyed on field ID (Iceberg, Delta) lose them silently.
Expected behavior
Take nullability and field IDs from the Catalyst schema. WriteJobDescription.dataColumns is already plumbed through to CometWriteFilesExec for column names (#5293), so the attribute list with its nullable flag and metadata is available at the same point — it needs to reach the native side as a serialized schema rather than just names.
Additional context
Found while reviewing #5293. Same family as #3425 (INT96) and #3427 (footer metadata keys): the written file is readable but is not byte-comparable with what Spark writes, which matters most for files other engines read — see #4658 and the plan to write Iceberg data files natively.
Describe the bug
ParquetWriterExecbuilds the output Parquet schema by renaming the fields of the incoming Arrow schema:So two pieces of schema information come from whatever the Comet scan/serde produced, not from the Catalyst schema Spark intends to write:
Nullability. Spark writes each column
requiredoroptionalaccording to the CatalystStructField.nullableof the target schema. Comet inherits the Arrow field's nullability, which can differ — a column Spark would markrequiredmay come outoptional, or vice versa. That changes the file's schema as seen by every reader, and a spuriouslyrequiredcolumn is the dangerous direction.Field IDs.
spark.sql.parquet.fieldId.write.enabledmakes Spark emit Parquet field IDs from the Catalyst metadata. Comet emits none, soParquetFieldIdIOSuite-style round trips and any consumer keyed on field ID (Iceberg, Delta) lose them silently.Expected behavior
Take nullability and field IDs from the Catalyst schema.
WriteJobDescription.dataColumnsis already plumbed through toCometWriteFilesExecfor column names (#5293), so the attribute list with itsnullableflag and metadata is available at the same point — it needs to reach the native side as a serialized schema rather than just names.Additional context
Found while reviewing #5293. Same family as #3425 (INT96) and #3427 (footer metadata keys): the written file is readable but is not byte-comparable with what Spark writes, which matters most for files other engines read — see #4658 and the plan to write Iceberg data files natively.