[SPARK-58330][SQL] Prevent silent dropping of dynamic options on same table references.

[anuragmantri]

What changes were proposed in this pull request?

This PR fixes dynamic table options being silently dropped when the same table is referenced more than once in a single statement. Options are now re-applied on a per-query relation cache hit, so each reference keeps its own options.

Why are the changes needed?

The per-query relation cache in RelationResolution is keyed by catalog, namespace, name, and time travel spec, but not by options. When the same table is referenced twice in one statement with different options, the second reference gets a cache hit and reuses the first reference's options, so the options the user wrote on the second reference are silently ignored. This affects all queries that self reference a table.

Does this PR introduce any user-facing change?

Yes. Options that were previously silently dropped on a second reference to the same table now take effect, and options that previously leaked from one reference onto another no longer do. This affects any statement with two or more references to one table.

Added a note to docs/sql-migration-guide.md scoped to SELECT/INSERT since those are the only two of the four statements that have shipped in a released version. UPDATE/MERGE self-reference support hasn't reached a release yet, so there's no upgrade path to document for those two.

How was this patch tested?

Added end-to-end tests, each asserting that both references keep their own options:

• DataSourceV2OptionSuite: INSERT selecting from the same table, a self-join, and a streaming self-join and streaming CTE (both analysis-only, no writeStream.start())
• MergeIntoTableSuiteBase: self-merge (target and source referencing the same table)
• UpdateTableSuiteBase: a subquery and a CTE referencing the same table as the update target
• DDLSuite: the same three shapes (self-join, INSERT self-select, CTE) for v1 (session-catalog/Hive) tables

Was this patch authored or co-authored using generative AI tooling?

I used Claude Code (Claude Opus 4.8) to generate the code and tests and verified manually.

[anuragmantri]

@dongjoon-hyun @peter-toth - Please provide your review on this.

[dongjoon-hyun]

+1, LGTM. Thank you, @anuragmantri .

[peter-toth]

Thanks for the PR, @anuragmantri!

This is the holistic fix for the limitation we recorded in #57227: AnalysisContext.relationCache is keyed by (catalog + namespace + name, timeTravelSpec) and not by options, so the second reference to a table in one statement got a cache hit and silently inherited the first reference's WITH (...) options. Re-applying each reference's own options to the cached plan, instead of adding options to the cache key, is the right shape: the table is still loaded once and both references share one Table instance, so they still see the same snapshot, and it matches what the two other cache-hit paths already do (adaptCachedRelation(cached, ref) at RelationResolution.scala:518-524 and the shared-relation-cache branch at :310). The details check out - finalOptions is clearWritePrivileges'd so the internal privileges key can't leak into the relation and from there into the write options V2Writes builds, the cache entry itself is left untouched, and the newInstance() / tag handling is unchanged. Three things below.

Blocking

• 1. "Does this PR introduce any user-facing change?" is answered No: it does introduce one. Options that were silently dropped now take effect, and options that leaked onto a second reference no longer do, so existing queries can behave differently. With a connector where a read option picks a snapshot/version, SELECT ... FROM t WITH (`snapshot-id` = X) a JOIN t b used to read snapshot X on both sides and now reads the live table for b - different results, not just a different split size. Please flip it to Yes with a sentence on the delta, and say whether you think it needs a docs/sql-migration-guide.md line. Same paragraph: "This affects INSERT, UPDATE, and MERGE" is narrower than the change - any statement with two references to one table is affected, plain SELECT included.

Non-blocking

• 2. v1 relations still share the first reference's options: applyOptions only rewrites DataSourceV2Relation, so a v1 (session-catalog data-source / Hive) table keeps the cached UnresolvedCatalogRelation's options - and v1 dynamic options are live and tested (DDLSuite.scala:1394 asserts SELECT * FROM t WITH ('delimiter' = ';') changes the rows for a CSV table). One extra case covers it; otherwise the description should scope the fix to DSv2. [inline: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala:370]
• 3. No test for two read references in one query: all three new tests pair a write reference with a read reference, so the base case of the bug - two reads with different options - is untested, and that is the case where the fix changes behavior in both directions. [inline: sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2OptionSuite.scala:124]

[anuragmantri]

Thanks for the thorough review, @peter-toth!

1. Updated the description to Yes and expanded to all statements that self-reference a table. I also added a section in migration guide scoped to SELECT and INSERT (only released operations with options).
2. Added the UnresolvedCatalogRelation case as you suggested, plus three new tests for v1 tables mirroring the DSv2 coverage
3. Added your self-join test as-is.

[peter-toth]

Re-checked through 6a7bfc4 — findings 2 and 3 resolved (the UnresolvedCatalogRelation case plus three v1 tests, and the read-read self-join test), nothing regressed. Finding 1 is half done: the answer is Yes now and the migration-guide note is right (SELECT = SPARK-36680 and INSERT = SPARK-49098 are the only two that shipped in a release; UPDATE/MERGE options are master-only, so 4.3 is the right version), but the title and the first two sections still scope the fix to DML.

Blocking

• 1. Description and title scope (round 1): the title says ... in DML, "What changes" says "in a single DML statement", and "Why are the changes needed" says "This affects INSERT, UPDATE, and MERGE" — but the fix sits in RelationResolution and applies to any statement with two references to one table. This PR's own migration note says SELECT, and two of the new tests (each reference in a self-join keeps its own dynamic options, self-reference to a v1 table keeps each reference's own dynamic options) are plain SELECTs. Please widen all three (dropping "in DML" from the title is enough there).

Non-blocking

• 4. Streaming v2 relations still keep the first reference's options (late catch): the case you added covers the streaming v1 shape as a side effect, but a v2 streaming table resolves to StreamingRelationV2, which applyOptions doesn't match, so a second FROM STREAM t WITH (...) reference still inherits the first's options. One more case line, or a note that streaming is out of scope.

Minor

• 5. Unstated invariant behind the unguarded transform (late catch): applyOptions rewrites every matching node with no identifier check, unlike adaptCachedRelation(cached, ref) at :521; that's safe only because every cache entry holds a single relation for its own identifier, which is worth one sentence in the method comment.

[peter-toth]

Finding 5. This transform rewrites every matching node in the cached plan, with no check that the node is the relation for the identifier being resolved — unlike the other cache-hit rewrite in this file, adaptCachedRelation(cached, ref) at :519-525, which only touches a relation passing matchesReference (same catalog and identifier).

It is safe today, but only because of an invariant that isn't written down anywhere: all three writers to relationCache (:313, :325, :493) store a plan with a single relation node, and that node belongs to the key. For a view the cached plan is SubqueryAlias(View(desc, child = parser.parseQuery(viewText))), i.e. the body is still unresolved when it goes into the cache, so it holds UnresolvedRelations rather than the two types matched here (same for MetricViewPlaceholder). If a cache entry ever held a resolved sub-plan — a view body resolved before caching, say — then SELECT ... FROM v WITH ('x' = 1) on a second reference would stamp the outer reference's options onto the relations of unrelated tables inside it, silently and with no error.

I'd just record the assumption rather than add a guard:

  /**
   * Re-applies `options` to the relation in a cached plan. Every `relationCache` entry holds a
   * single relation for its own identifier (a view's body is still unresolved when it is cached),
   * so this cannot reach another table's relation.
   */
  private def applyOptions(

An identifier guard mirroring matchesReference is the tempting move but it's awkward for UnresolvedCatalogRelation, which only carries a CatalogTable — you'd be comparing a qualified TableIdentifier against the v2 Identifier plus catalog.name, and getting that slightly wrong means options quietly stop being re-applied, i.e. this PR's bug comes back on some path. If you do want a structural guard instead of the comment, matching the shapes (case SubqueryAlias(id, r: DataSourceV2Relation) => ..., case r: DataSourceV2Relation => ..., case other => other) keeps it to the top relation without any name comparison.

[anuragmantri]

I added the suggested comment on the method.

[peter-toth]

Finding 4. This line also fixes the streaming v1 path, since createRelation wraps a session-catalog streaming table as SubqueryAlias(_, UnresolvedCatalogRelation(v1Table, options, isStreaming = true)) (RelationResolution.scala:398-401). The v2 streaming shape is the one left out: createRelation builds StreamingRelationV2(..., options, ...) (:438-450), a LeafNode that holds its options in extraOptions, which applyOptions doesn't match — so on a cache hit the second reference keeps the first reference's options.

It is reachable from SQL: optionsClause is allowed on a stream relation (SqlBaseParser.g4:447-448, AstBuilder.visitStreamTableName at AstBuilder.scala:3224) and a stream-stream self join is expressible, e.g. SELECT * FROM STREAM t WITH ('a'='1') x JOIN STREAM t WITH ('a'='2') y ON x.id = y.id (cf. StreamRelationSuite.scala:58). StreamingRelationV2 is already imported at :31, so it's one line:

Suggested change

	case r: UnresolvedCatalogRelation => r.copy(options = options)
	case r: UnresolvedCatalogRelation => r.copy(options = options)
	case r: StreamingRelationV2 => r.copy(extraOptions = options)

If you'd rather keep streaming out of this PR, a sentence in the method comment saying the streaming v2 relation is not covered would do — this PR removes the only comment that recorded the old limitation.

[anuragmantri]

I added StreamingRelationV2 case and added tests.

[anuragmantri]

Thanks, @peter-toth!

1. Removed DML wording from the PR description.

2. Added the StreamingRelationV2 case (extraOptions), plus two tests: a streaming self-join and a streaming CTE self-reference, both analysis-only (never call writeStream.start(), so no actual streaming query runs) — verified each fails without the fix and passes with it.

3. Added a doc comment on applyOptions stating the invariant: every relationCache entry holds a single relation for its own identifier (a view's body is still unresolved when cached), so the unguarded transform can't reach another table's relation.

[anuragmantri]

I added the suggested comment on the method.

[anuragmantri]

I added StreamingRelationV2 case and added tests.

[peter-toth]

@anuragmantri, can you please update the PR with the latest master? Unfortunately my #57563 caused some conflicts.