feat(spark): unify Databricks PAT, OAuth M2M, connector OIDC, and namespace OIDC auth

[rohitrsh]

Final integration PR — merge this PR only.

This branch contains all four separately reviewed and approved Databricks authentication commits from #3452, #3453, #3454, and this PR. After this PR merges, #3452–#3454 can be closed as included here.

Summary

• Introduces a Databricks authentication strategy boundary while preserving PAT as the default.
• Adds OAuth M2M authentication using short-lived service-principal tokens.
• Adds connector workload OIDC federation using projected JWT exchange.
• Adds workflow-namespace ServiceAccount OIDC federation through Kubernetes TokenRequest.
• Persists authentication metadata and refreshes short-lived tokens after HTTP 401 responses.
• Documents configuration, workload identity requirements, and namespace ServiceAccount RBAC.

Review slices:

1. refactor(spark): isolate Databricks PAT authentication #3452 — PAT strategy refactor
2. feat(spark): add Databricks OAuth M2M authentication #3453 — OAuth M2M authentication
3. feat(spark): add Databricks OIDC connector authentication #3454 — Connector workload OIDC federation
4. Namespace ServiceAccount OIDC federation (this PR)

Test plan

• Rebased all four commits onto current upstream master
• Focused Databricks connector suite — 85 passed
• Full Spark plugin suite in Linux, Python 3.12, and OpenJDK 17 — 110 passed
• make lint
• Ruff lint and format hooks run twice on changed Python files
• git diff --check
• Spark plugin sdist/wheel build and import smoke test
• Changed-file, full-patch, commit-metadata, confidentiality, secret-pattern, and attribution audits

This PR is the single merge target for the complete four-part Databricks authentication stack. Hosted CI must complete before merge.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.65%. Comparing base (5f18ae7) to head (24ffcd2).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3455      +/-   ##
==========================================
+ Coverage   76.27%   77.65%   +1.37%     
==========================================
  Files         217      252      +35     
  Lines       22943    24332    +1389     
  Branches     3036     3036              
==========================================
+ Hits        17499    18894    +1395     
- Misses       4556     4613      +57     
+ Partials      888      825      -63     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[rohitrsh]

@pingsutw This PR is now the final integration merge target, following the same approach used for the SageMaker stack in #3459.

It contains all four approved Databricks authentication commits from #3452, #3453, #3454, and #3455, rebased onto current master. All local verification and the complete hosted CI workflow pass.

Please merge #3455 only. After it merges, I will close #3452–#3454 as included in this PR.