[CONNECT] Skip redundant DSv2 table refreshes in Spark Connect

[longvu-db]

What changes were proposed in this pull request?

This PR implements the server-side SQL configuration approach suggested in #57043 (comment).

It adds the internal spark.sql.execution.skipV2TableRefresh configuration, defaulting to false, and sets it to true whenever the Spark Connect server creates or clones a session. QueryExecution uses this setting when deriving its effective refreshPhaseEnabled value, so the optimization covers every Connect path that creates a query execution, including SQL, DataFrame operations, checkpoint, local checkpoint, and future call sites.

Spark Connect's plan cache is handled conservatively. A cache hit can contain an already-resolved DataSourceV2Relation, for which analysis alone cannot reload current table metadata. Such cached plans are cloned and tagged so their QueryExecution retains the refresh phase. Freshly analyzed Connect plans still skip it.

The existing QueryExecution constructor signature and refreshPhaseEnabled accessor remain binary-compatible.

Why are the changes needed?

Spark Connect normally analyzes a plan immediately before execution. The DSv2 table refresh phase then performs another catalog loadTable request without changing the outcome, adding an avoidable catalog round trip and latency.

Using spark.api.mode on the server is not reliable because it is a client-side builder/launcher directive and is not propagated to Spark Connect server sessions. A server-side session setting explicitly records the execution environment and avoids having to update every Connect call site.

The plan-cache exception preserves correctness for resolved DSv2 relations that may outlive the table metadata captured during their original analysis.

Does this PR introduce any user-facing change?

No. The new configuration is internal. Spark Classic keeps the refresh phase enabled, while Spark Connect avoids redundant catalog refresh requests for freshly analyzed plans.

How was this patch tested?

Added a counting DSv2 catalog and tests that assert the exact number of loadTable calls, not only query results. Coverage includes:

• Classic defaults and explicit refresh skipping.
• Connect-created sessions and cloned sessions.
• DataFrame table, drop, SQL, checkpoint, and local checkpoint paths.
• Repeated execution through the Connect plan cache, where cached resolved DSv2 plans retain the refresh.
• Existing Connect DSv2 freshness scenarios.

The following suites passed (119 tests total):

• QueryExecutionSuite: 29 tests.
• SparkConnectSessionManagerSuite: 11 tests.
• SparkConnectCloneSessionSuite: 9 tests.
• SparkConnectServerTestSuite: 21 tests.
• DataSourceV2DataFrameConnectSuite: 49 tests.

Compilation passed for catalyst/Test/compile, sql/Test/compile, and connect/Test/compile. Scalastyle passed for the catalyst, SQL, and Connect modules.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex