[WIP][INFRA] Add smart class-level test selection for post-merge CI

[zhengruifeng]

What changes were proposed in this pull request?

This draft adds a dedicated smart class-level post-merge test-selection workflow without changing the existing build_and_test.yml workflow.

For trusted Apache Spark pushes, with this draft branch temporarily allowed in the personal fork for verification, the workflow uses Copilot CLI according to the Copilot CLI Actions guidance. The CLI receives a separate, read-only prompt for the pushed tip commit. It analyzes the actual diff, class hierarchies, and test call sites, and considers runnable PySpark doctests for changed docstrings.

The helper validates every proposed target against the checkout before execution, accepts only Scala or Java JVM suites and importable PySpark test modules, excludes R tests, and retains at most 20 targets of each type. A precompile job builds Spark once when targets were selected. The dependent JVM and Python jobs run in parallel and execute each selected class or module individually. The JVM runner resolves each suite to its SBT test project and invokes testOnly; the Python runner uses python/run-tests --testnames.

Why are the changes needed?

Post-merge changes benefit from quick, focused feedback in addition to the existing deterministic CI. Selecting validated class-level tests by dependency, hierarchy, and call-site relevance reduces unrelated test execution while retaining the existing build and test workflows as independent coverage.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

• Ran conda run -n spark-dev-313 ./dev/lint-python.
• Ran git diff --check.
• Validated representative JVM, Java, PySpark, and doctest targets with dev/smart_test_selection.py.
• Verified the JVM runner command construction with a mocked subprocess.
• No full Spark test suite was run; the dedicated GitHub Actions workflow is being verified on this draft branch.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Codex (GPT-5)