[SPARK-57197][SQL][FOLLOWUP] Improve test coverage for merge into schema evolution#56242
[SPARK-57197][SQL][FOLLOWUP] Improve test coverage for merge into schema evolution#56242jiwen624 wants to merge 1 commit into
Conversation
jiwen624
changed the title
|
@szehon-ho Could you review this follow-up PR improving the test coverage of MERGE INTO schema evolution? I’m reusing the Jira for this PR but please let me know if it needs to be under a separate one 🙇 |
szehon-ho
left a comment
szehon-ho
left a comment
There was a problem hiding this comment.
Yes let's make a new JIRA. Could you also add a test for missing source field with special char (and nested as well)
| } | ||
|
|
||
| for (colName <- Seq("job.title", "job title")) { | ||
| testEvolution( |
There was a problem hiding this comment.
nit: could we do this for nested struct as well?
jiwen624
changed the title
jiwen624
force-pushed
the
spark-56462-followup-test-coverage
branch
2 times, most recently
from
b97ab13 to
bae9b86
Compare
|
@szehon-ho I added test cases per your comments. The build is currently failing because of the issue being fixed by this PR (still open) and another flaky test. I'll rebase and rebuild it once that fix is merged. |
szehon-ho
left a comment
szehon-ho
left a comment
There was a problem hiding this comment.
thanks, just had two style nits
|
|
||
| for (colName <- Seq("job.title", "job title")) { | ||
| testEvolution( | ||
| s"SPARK-57197: target has special-char column missing from source: $colName")( |
There was a problem hiding this comment.
nit: can we remove the SPARK JIRA from these tests? its not very helpful
|
|
||
| for (subFieldName <- Seq("job.title", "job title")) { | ||
| testNestedStructsEvolution( | ||
| s"SPARK-57197: source struct missing nested field with special-char name: $subFieldName")( |
There was a problem hiding this comment.
can we harmonize the description with the previous (ie, target has ... missing from source')
jiwen624
force-pushed
the
spark-56462-followup-test-coverage
branch
from
bae9b86 to
fc9baf8
Compare
2 participants
What changes were proposed in this pull request?
This is a follow-up PR from the comment #55329 (comment) - to add test coverage for MERGE INTO schema evolution:
Why are the changes needed?
To increase test coverage of MERGE INTO schema evolution.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Added test cases
Was this patch authored or co-authored using generative AI tooling?
Yes. Claude Code