[SPARK-58319][SDP] Derive SCD2 AutoCDC target table schema

[anew]

Title: [SPARK-58319][SDP] Derive SCD2 AutoCDC target table schema

Body:

What changes were proposed in this pull request?

Implement the ScdType.Type2 branch of AutoCdcMergeFlow.schema and load in Flow.scala, which previously threwAUTOCDC_SCD2_NOT_SUPPORTED. The SCD2 target schema is the user-selected columns followed by the SCD2 framework columns, in the exact order and nullability that Scd2BatchProcessor.preprocessMicrobatch projects and the merges persist:

• __START_AT (typed as the sequencing type, nullable)
• __END_AT (typed as the sequencing type, nullable)
• _cdc_metadata (the SCD2 operational metadata struct, non-null)

To build this from the graph package, the visibility of a few Scd2BatchProcessor members is widened: startAtColName / endAtColName (private[autocdc] -> private[pipelines]) and constructCdcMetadataCol (private -> private[pipelines]).cdcMetadataColSchema was already private[pipelines].

This is one subtask of enabling SCD2 AutoCDC end to end (under SPARK-56249). After this change, SCD2 AutoCDC flows are constructible (their output schema resolves), but they still cannot run: the auxiliary-table derivation (AutoCdcAuxiliaryTable) and the flow planner (FlowPlanner) still reject SCD2. Those remaining gates and the streaming-write wiring are handled in follow-up subtasks (SPARK-58320, SPARK-58321).

Why are the changes needed?

Deriving the SCD2 target schema is a prerequisite for SCD2 support: the pipeline's downstream dataset materialization and schema evolution consume AutoCdcMergeFlow.schema, so an SCD2 flow cannot be registered or its target table created until this branch produces the correct augmented schema. The Scd2BatchProcessor reconciliation engine that produces rows in this exact shape already landed (SPARK-57378); this connects the flow's declared schema to it.

Does this PR introduce any user-facing change?

No. SCD2 AutoCDC flows remain unusable end to end (still gated by the auxiliary-table and planner checks), so there is no change to released or user-facing behavior. This only makes the flow's schema resolvable internally.

How was this patch tested?

New unit tests in AutoCdcFlowSuite (replacing the obsolete "rejects SCD2 at construction" test):

• schema appends the SCD2 framework columns in the correct order/nullability when no column selection is set;
• a columnSelection narrows only the user-data portion, before the framework columns;
• the framework columns and the metadata record-start-at field pick up a non-Long resolved sequencing type;
• load()'s empty-dataframe schema matches schema for SCD2.

Was this patch authored or co-authored using generative AI tooling?

Generated-By: Claude Opus 4.8