create Spark 4.1 and 4.2 support and fix unit tests

[ukby1234]

Created support for Spark 4.1 and Spark 4.2.
One unit tests failed because the generated analyzed plan fails the validation. This is the same plan generated for Spark 3.5 as well. Spark introduced a feature to reconcile schema for different datasets, and changed the unit to address the column-ordering issue.
Also tested with a locally published version against https://github.com/scalapb/sparksql-scalapb, and all tests there are passing as well.

[ukby1234]

@pomadchin can i get a review from the maintainer? thanks.

[pomadchin]

@ukby1234 ty for the PR! Also that's your first contribution! 🎉

One unit tests failed because the generated analyzed plan fails the validation

What tests did fail? 👀

[pomadchin]

LGTM, could I ask to update REAMDE.md as well if possible?
Will merge once the question about the test is cleared.

[ukby1234]

LGTM, could I ask to update REAMDE.md as well if possible? Will merge once the question about the test is cleared.

You mean update the supported versions?

[pomadchin]

LGTM, could I ask to update REAMDE.md as well if possible? Will merge once the question about the test is cleared.

You mean update the supported versions?

yes!

[ukby1234]

Ok the test failed because analyzer.checkAnalysis failed. The dummyPlan generated above uses GetColumnByOrdinal but the analyzed plan uses GetStructField. In Spark 3 and Spark 4, both versions generated analyzedPlan like getting struct field a.a.a where a.a is a double type. In theory, it should generate b.a.a.
In Spark 4.1+, they added data type check for GetStructField, which resulted in data type mismatch. I don't think it's a regression since Spark 4.1+ because the generated plan never works as intended, and folks with DataFrame columns out of order should use Spark's native .to(schema) feature to reorder the columns.
@pomadchin

[ukby1234]

Will update the README.md to update the supported versions. @pomadchin

[ukby1234]

ok updated README.md @pomadchin

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.23%. Comparing base (bc1a19f) to head (d99c075).
⚠️ Report is 14 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #966      +/-   ##
==========================================
+ Coverage   96.21%   96.23%   +0.01%     
==========================================
  Files          66       68       +2     
  Lines        1296     1328      +32     
  Branches       48       48              
==========================================
+ Hits         1247     1278      +31     
- Misses         49       50       +1     

Flag	Coverage Δ
2.13-root-spark35	96.26% <ø> (?)
2.13-root-spark40	?
2.13-root-spark41	96.21% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.