[infra] Derive and validate the Flink versions tools/ut.sh uses

[weiqingy]

Linked issue: #1006

Purpose of change

tools/ut.sh told the reader that omitting -f tests every supported Flink version. It tests one, and on the Java unit-test path the selected version is not used at all. Because DEFAULT_FLINK_VERSION was pinned to 2.2 while the root pom carries 2.3.0, a bare ./tools/ut.sh tested Java on 2.3.0 and Python on 2.2.x in the same command.

The first commit derives the default from <flink.version> in the root pom instead of pinning it, so both lanes land on the same version and the value cannot drift again. That drift is the root cause rather than an accident: tools/update_flink_version.sh rewrites poms only and never touches this script, so the pinned literal fell a minor line behind. The derived value is validated to be shaped x.y before use, because it flows unvalidated into dist/flink-<v> paths, -Pflink-<v> profile names and the apache-flink~=<v>.0 requirement. Reading the property with sed follows tools/update_flink_version.sh, which already reads the same property the same way.

The second commit warns when -f cannot affect what was selected. ./tools/ut.sh -j -f 1.20 previously accepted the flag and then tested exactly what it would have anyway. A warning was chosen over rejecting the flag, since rejecting it turns a currently-working command into a failing one.

The third commit validates the versions -f accepts. It took any string, so -f 9.9 asked Maven to build dist/flink-9.9, a module that does not exist, and a typo looked like it worked. The supported set is derived from the dist/flink-* modules rather than a second hardcoded list, and the derived default is checked against it too, so the two derivations cannot drift apart. It also adds a flink-2.3 profile to the e2e integration pom: that pom defined flink-1.20 through flink-2.2 while 2.3 was its unprofiled default, so -f 2.3 passed a profile Maven warned about and ignored, landing on 2.3.0 by fallthrough.

The new profile deliberately omits flink.log4j2.version. Its four siblings pin 2.24.3 because they need a value different from the parent's, while 2.3 needs the parent's 2.25.3, so a redundant pin would stop tracking it on the next bump. Verified behavior-neutral: flink.version, flink.agents.dist.artifactId and flink.log4j2.version all resolve identically with and without -Pflink-2.3.

This subsumes #1008, which was filed for those two defects and then closed as completed once the work moved here.

The help text now states the derived default and the flag's real scope, and its supported-version list is derived rather than a literal.

One option in the issue is deliberately not taken. Making the default iterate every supported version is not implementable today: runtime fails test-compile at 1.20 with 5 errors in RescalingTest.java, which imports org.apache.flink.streaming.api.functions.sink.legacy and ...source.legacy, packages that exist only in Flink 2.x. It would also multiply every job that runs the Java unit tests by roughly five, to duplicate coverage the java_it_tests matrix already provides across all five versions.

Tests

New tools/test/unit/ut_sh_version_default.bats (8 cases) and tools/test/unit/ut_sh_flink_flag_scope.bats (6 cases), in the existing bats harness that CI already runs. Both use the subprocess and shim patterns from tools/test/integration/help.bats and dry_run.bats; neither sources ut.sh, which has no run guard.

The version cases cover the derived default, that it follows a changed pom rather than coincidentally matching today's value, whitespace-padded XML, and the three malformed shapes that must be fatal. The scope cases pin the firing condition rather than just the message, so an over-broad or under-broad condition fails.

A third file covers validation: that an unsupported -f is rejected, that the supported set follows a fake dist/ rather than being pinned to today's tree, and that each rejection names its own cause. One case is a drift guard asserting the three lists that must agree still do — the dist/flink-* directories, dist/pom.xml's <modules>, and the e2e integration pom's flink-* profile ids. Their disagreement is what left 2.3 without a profile in the first place.

Full harness: 212 ok / 1 not ok. The single failure is pre-existing and unrelated, in verify_example_job.bats via e2e-test/test-scripts/test_submit_examples_to_flink.sh:362 (remaining_pids[@]: unbound variable); it reproduces identically on unmodified main.

./tools/ut.sh -p was run on the new default: 825 passed, 13 skipped, exit 0, against apache-flink 2.3.0 confirmed from the environment rather than the log.

Note one behavior change: the Python unit-test jobs in ci.yml and build_wheel.yml now install apache-flink~=2.3.0 rather than ~=2.2.0. PyFlink 2.3.0 is published and is the current latest on PyPI. Both integration matrices pass -e -f explicitly and are unaffected, and none of the six workflow invocations emits the new warning.

Not covered: the pip fallback branch of ut.sh -p was not exercised on the new default, only the uv branch that CI uses.

API

No public API change. The observable change is to the script's CLI behavior and help text, described above.

Documentation

• doc-needed
• doc-not-needed
• doc-included

Every inaccurate claim was in the script's own help text, which this PR corrects. No file under docs/ references tools/ut.sh.

Was this patch authored or co-authored using generative AI tooling?

• Yes
• No

Generated-by: Claude Code 2.1.229 (Claude Opus 5)