What is the problem the feature request solves?
WriteTaskStatsTracker.newRow(filePath: String, row: InternalRow) is a per-row callback. Comet's native write path has columnar batches, not InternalRows, so CometWriteFilesExec.recordRows calls it n times with InternalRow.empty rather than materializing every row just to hand it straight back.
That is exactly right for BasicWriteTaskStatsTracker, the only implementation Spark ships, which ignores the row argument and just increments a counter (BasicWriteStatsTracker.scala). But a third-party tracker that inspects row contents — a Delta or custom-catalog stats collector, say — would silently compute its statistics over empty rows.
Today this logs a warning per non-BasicWriteTaskStatsTracker instance. A warning is the honest minimum, but it is not a guarantee.
Describe the potential solution
The obstacle is that a plan-time guard is not possible: WriteJobDescription.statsTrackers only exists at execution time, by which point getSupportLevel has already accepted the write and there is no way to fall back.
Options:
- Materialize rows only when a non-basic tracker is present — correct, and pays the row-conversion cost solely in the case that needs it.
- Fail the write with a clear message instead of warning, so nobody gets wrong statistics silently.
- Find a plan-time signal for the tracker set so the write can fall back to Spark gracefully.
Option 1 is the most useful; option 2 is a smaller step if the row-conversion path is not worth building yet.
Additional context
Introduced by #5293, which moved native writes onto Spark's WriteFilesExec seam and therefore onto Spark's stats-tracker contract. The old path bypassed the trackers entirely and reported its own metrics, so this is a new obligation rather than a regression.
What is the problem the feature request solves?
WriteTaskStatsTracker.newRow(filePath: String, row: InternalRow)is a per-row callback. Comet's native write path has columnar batches, notInternalRows, soCometWriteFilesExec.recordRowscalls itntimes withInternalRow.emptyrather than materializing every row just to hand it straight back.That is exactly right for
BasicWriteTaskStatsTracker, the only implementation Spark ships, which ignores the row argument and just increments a counter (BasicWriteStatsTracker.scala). But a third-party tracker that inspects row contents — a Delta or custom-catalog stats collector, say — would silently compute its statistics over empty rows.Today this logs a warning per non-
BasicWriteTaskStatsTrackerinstance. A warning is the honest minimum, but it is not a guarantee.Describe the potential solution
The obstacle is that a plan-time guard is not possible:
WriteJobDescription.statsTrackersonly exists at execution time, by which pointgetSupportLevelhas already accepted the write and there is no way to fall back.Options:
Option 1 is the most useful; option 2 is a smaller step if the row-conversion path is not worth building yet.
Additional context
Introduced by #5293, which moved native writes onto Spark's
WriteFilesExecseam and therefore onto Spark's stats-tracker contract. The old path bypassed the trackers entirely and reported its own metrics, so this is a new obligation rather than a regression.