Skip to content

[ExecuTorch][WebGPU] Convert remaining native tests to GTest#20706

Open
JulianCloudNTH wants to merge 4 commits into
gh/JulianCloudNTH/85/basefrom
gh/JulianCloudNTH/85/head
Open

[ExecuTorch][WebGPU] Convert remaining native tests to GTest#20706
JulianCloudNTH wants to merge 4 commits into
gh/JulianCloudNTH/85/basefrom
gh/JulianCloudNTH/85/head

Conversation

@JulianCloudNTH

@JulianCloudNTH JulianCloudNTH commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Stack from ghstack (oldest at bottom):

Convert the remaining hand-rolled int main() + printf/bool ok native tests to GTest so the whole backends/webgpu/test/ suite is uniform, filterable via --gtest_filter, and self-reporting (extends the GTest conversion already applied to test_dynamic_shape). The five converted files are a harness-only change — every test case, tensor shape, tolerance, artifact filename, and skip condition is preserved 1:1, only the pass/fail reporting mechanism changes — and this diff additionally wires the already-GTest webgpu_dynamic_shape_test into the CI runner so the dynamic-shape suite actually executes.

Key changes:

  • test/test_webgpu_native.cpp, test/native/test_dispatch_order.cpp, test/native/test_index.cpp, test/native/test_scratch_buffer.cpp, test/native/test_update_cache.cppmain+printf/bool ok accumulator → TEST() cases using EXPECT_*/ASSERT_*; each keeps a custom main() that brings up the WebGPU device once then RUN_ALL_TESTS() (device-absent still SKIPs by returning 0). test_index/test_webgpu_native use inclusive EXPECT_LE(err, tol) to match the original err > tol fail gate exactly.
  • CMakeLists.txt — move every native-test target into the if(TARGET GTest::gtest) block, linking GTest::gtest.
  • scripts/test_webgpu_native_ci.sh — add -DEXECUTORCH_BUILD_TESTS=ON to the native-test configure so the now-gtest-gated targets are defined, and wire webgpu_dynamic_shape_test into the runner: export its .ptes + goldens via export_dynamic_shape_cases, add it to the built/run target list behind the same --target help probe, and run it guarded (mirroring the index test).
  • test/test_build_webgpu.sh — add -DEXECUTORCH_BUILD_TESTS=ON so the local build script (which builds the now-gtest-gated targets unconditionally) still finds them.
    @exported-using-ghexport

Differential Revision: D110536636

Differential Revision: D110536636

[ghstack-poisoned]
@pytorch-bot

pytorch-bot Bot commented Jul 2, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20706

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit 004e1d9 with merge base 124b677 (image):

NEW FAILURE - The following job has failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 2, 2026
@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@JulianCloudNTH

Copy link
Copy Markdown
Contributor Author

@claude Review thoroughly. Check for stale code / bugs.

@claude

claude Bot commented Jul 3, 2026

Copy link
Copy Markdown

Claude finished @JulianCloudNTH's task in 8m 7s —— View job


Review: WebGPU native tests → GTest

Reviewed all 7 files against origin/gh/JulianCloudNTH/85/base. The conversion is careful and faithful — cases, shapes, tolerances, artifact names, skip conditions, and the device-absent→return 0 SKIP behavior are preserved, and every deleted sweep wrapper maps cleanly onto a TEST(). ASSERT_* inside the void helpers is the correct pattern here (it returns from the helper, matching the old return false). One real bug and a couple of minor notes below.


🔴 Sibling build script will break: test_build_webgpu.sh still lacks -DEXECUTORCH_BUILD_TESTS=ON

This PR moves webgpu_native_test, webgpu_dispatch_order_test, webgpu_scratch_buffer_test, webgpu_update_cache_test, webgpu_index_test inside the if(TARGET GTest::gtest) block (CMakeLists.txt:190-209). GTest::gtest only exists when googletest is added as a subdirectory, which happens only under BUILD_TESTING/EXECUTORCH_BUILD_TESTS:

third-party/CMakeLists.txt:18   if(BUILD_TESTING) add_subdirectory(googletest) endif()
CMakeLists.txt:253              if(EXECUTORCH_BUILD_TESTS) include(CTest) else() set(BUILD_TESTING OFF)

You correctly added -DEXECUTORCH_BUILD_TESTS=ON to scripts/test_webgpu_native_ci.sh. But the sibling backends/webgpu/test/test_build_webgpu.sh — the documented local end-to-end script — configures without it (test_build_webgpu.sh:84-95) and then builds these targets unconditionally:

cmake --build "${NATIVE_BUILD_DIR}" --target webgpu_native_test -j${NPROC}          # :97
cmake --build "${NATIVE_BUILD_DIR}" --target webgpu_dispatch_order_test -j${NPROC}  # :98
cmake --build "${NATIVE_BUILD_DIR}" --target webgpu_scratch_buffer_test -j${NPROC}  # :99

Unlike the CI script, this one does not probe --target help first, so if no system GTest is picked up by the find_package(GTest QUIET) fallback, these targets won't be defined and the build fails with "unknown target". Before this PR the targets were defined regardless of EXECUTORCH_BUILD_TESTS, so the script worked. Add -DEXECUTORCH_BUILD_TESTS=ON to the configure at test_build_webgpu.sh:84. Fix this →


🟡 Minor: pass/fail boundary changed from inclusive to exclusive in a few tests

Several gates moved from if (err > tol) FAIL (pass iff <= tol) to EXPECT_LT(err, tol) (pass iff < tol), e.g. test_index.cpp (EXPECT_LT(max_abs_err, 1e-3f) / EXPECT_LT(max_rel_err, 1e-3f)), test_update_cache.cpp / test_webgpu_native.cpp update_cache (EXPECT_LT(max_abs_err, 1e-3f)). The AND/OR logic is preserved; only the exact-boundary case (err == 1e-3f) flips from pass to fail. This is essentially never hit in practice and is stricter, not looser — but it's a deviation from the "tolerance preserved 1:1" claim. dispatch_order avoided this by keeping the exact expression via EXPECT_FALSE(a && b); the others could do the same with EXPECT_LE if strict 1:1 is desired. Non-blocking.


🟢 Things I checked that are correct

  • test_query_pool_roundtrip(*get_default_webgpu_context()) — safe: RUN_ALL_TESTS() only runs after set_default_webgpu_context(&ctx), and the device-absent path returns before it.
  • SDPA expect_reject path (ASSERT_NE(err, Error::Ok) + return) preserves the original PASS/FAIL semantics.
  • SdpaAllFamiliesRanWhenDirSet recomputes .pte presence independently but is equivalent to the old sdpa_ran && replay_ran && dyn_ran && incache_ran guard.
  • required/heavy gating in the q4gsw & SDPA sweeps correctly maps FAILADD_FAILURE() (non-fatal, so the loop continues, matching the old ok = ... && ok) and SKIPprintf/GTEST_SKIP.
  • quant_within_tol moved to a full definition (forward-decl removed) with identical body; no other refs dangling.
  • CI script --target help probe still guards the CI path, so the webgpu_index_test move into the gated block is covered there.

Nice, mechanical, well-commented conversion overall — the one thing to fix before landing is the test_build_webgpu.sh configure flag.
gh/JulianCloudNTH/85/head

[ghstack-poisoned]
JulianCloudNTH added a commit that referenced this pull request Jul 3, 2026
Pull Request resolved: #20706

Convert the remaining hand-rolled `int main()` + printf/`bool ok` native tests to GTest so the whole `backends/webgpu/test/` suite is uniform, filterable via `--gtest_filter`, and self-reporting (extends the GTest conversion already applied to `test_dynamic_shape`). The five converted files are a harness-only change — every test case, tensor shape, tolerance, artifact filename, and skip condition is preserved 1:1, only the pass/fail reporting mechanism changes — and this diff additionally wires the already-GTest `webgpu_dynamic_shape_test` into the CI runner so the dynamic-shape suite actually executes.

Key changes:
- `test/test_webgpu_native.cpp`, `test/native/test_dispatch_order.cpp`, `test/native/test_index.cpp`, `test/native/test_scratch_buffer.cpp`, `test/native/test_update_cache.cpp` — `main`+`printf`/`bool ok` accumulator → `TEST()` cases using `EXPECT_*`/`ASSERT_*`; each keeps a custom `main()` that brings up the WebGPU device once then `RUN_ALL_TESTS()` (device-absent still SKIPs by returning 0). `test_index`/`test_webgpu_native` use inclusive `EXPECT_LE(err, tol)` to match the original `err > tol` fail gate exactly.
- `CMakeLists.txt` — move every native-test target into the `if(TARGET GTest::gtest)` block, linking `GTest::gtest`.
- `scripts/test_webgpu_native_ci.sh` — add `-DEXECUTORCH_BUILD_TESTS=ON` to the native-test configure so the now-gtest-gated targets are defined, and wire `webgpu_dynamic_shape_test` into the runner: export its `.pte`s + goldens via `export_dynamic_shape_cases`, add it to the built/run target list behind the same `--target help` probe, and run it guarded (mirroring the `index` test).
- `test/test_build_webgpu.sh` — add `-DEXECUTORCH_BUILD_TESTS=ON` so the local build script (which builds the now-gtest-gated targets unconditionally) still finds them.
ghstack-source-id: 399790083
@exported-using-ghexport

Differential Revision: [D110536636](https://our.internmc.facebook.com/intern/diff/D110536636/)
[ghstack-poisoned]
JulianCloudNTH added a commit that referenced this pull request Jul 3, 2026
Pull Request resolved: #20706

Convert the remaining hand-rolled `int main()` + printf/`bool ok` native tests to GTest so the whole `backends/webgpu/test/` suite is uniform, filterable via `--gtest_filter`, and self-reporting (extends the GTest conversion already applied to `test_dynamic_shape`). The five converted files are a harness-only change — every test case, tensor shape, tolerance, artifact filename, and skip condition is preserved 1:1, only the pass/fail reporting mechanism changes — and this diff additionally wires the already-GTest `webgpu_dynamic_shape_test` into the CI runner so the dynamic-shape suite actually executes.

Key changes:
- `test/test_webgpu_native.cpp`, `test/native/test_dispatch_order.cpp`, `test/native/test_index.cpp`, `test/native/test_scratch_buffer.cpp`, `test/native/test_update_cache.cpp` — `main`+`printf`/`bool ok` accumulator → `TEST()` cases using `EXPECT_*`/`ASSERT_*`; each keeps a custom `main()` that brings up the WebGPU device once then `RUN_ALL_TESTS()` (device-absent still SKIPs by returning 0). `test_index`/`test_webgpu_native` use inclusive `EXPECT_LE(err, tol)` to match the original `err > tol` fail gate exactly.
- `CMakeLists.txt` — move every native-test target into the `if(TARGET GTest::gtest)` block, linking `GTest::gtest`.
- `scripts/test_webgpu_native_ci.sh` — add `-DEXECUTORCH_BUILD_TESTS=ON` to the native-test configure so the now-gtest-gated targets are defined, and wire `webgpu_dynamic_shape_test` into the runner: export its `.pte`s + goldens via `export_dynamic_shape_cases`, add it to the built/run target list behind the same `--target help` probe, and run it guarded (mirroring the `index` test).
- `test/test_build_webgpu.sh` — add `-DEXECUTORCH_BUILD_TESTS=ON` so the local build script (which builds the now-gtest-gated targets unconditionally) still finds them.
ghstack-source-id: 399809713
@exported-using-ghexport

Differential Revision: [D110536636](https://our.internmc.facebook.com/intern/diff/D110536636/)
@JulianCloudNTH JulianCloudNTH requested a review from psiddh July 3, 2026 21:27
[ghstack-poisoned]
JulianCloudNTH added a commit that referenced this pull request Jul 3, 2026
Pull Request resolved: #20706

Convert the remaining hand-rolled `int main()` + printf/`bool ok` native tests to GTest so the whole `backends/webgpu/test/` suite is uniform, filterable via `--gtest_filter`, and self-reporting (extends the GTest conversion already applied to `test_dynamic_shape`). The five converted files are a harness-only change — every test case, tensor shape, tolerance, artifact filename, and skip condition is preserved 1:1, only the pass/fail reporting mechanism changes — and this diff additionally wires the already-GTest `webgpu_dynamic_shape_test` into the CI runner so the dynamic-shape suite actually executes.

Key changes:
- `test/test_webgpu_native.cpp`, `test/native/test_dispatch_order.cpp`, `test/native/test_index.cpp`, `test/native/test_scratch_buffer.cpp`, `test/native/test_update_cache.cpp` — `main`+`printf`/`bool ok` accumulator → `TEST()` cases using `EXPECT_*`/`ASSERT_*`; each keeps a custom `main()` that brings up the WebGPU device once then `RUN_ALL_TESTS()` (device-absent still SKIPs by returning 0). `test_index`/`test_webgpu_native` use inclusive `EXPECT_LE(err, tol)` to match the original `err > tol` fail gate exactly.
- `CMakeLists.txt` — move every native-test target into the `if(TARGET GTest::gtest)` block, linking `GTest::gtest`.
- `scripts/test_webgpu_native_ci.sh` — add `-DEXECUTORCH_BUILD_TESTS=ON` to the native-test configure so the now-gtest-gated targets are defined, and wire `webgpu_dynamic_shape_test` into the runner: export its `.pte`s + goldens via `export_dynamic_shape_cases`, add it to the built/run target list behind the same `--target help` probe, and run it guarded (mirroring the `index` test).
- `test/test_build_webgpu.sh` — add `-DEXECUTORCH_BUILD_TESTS=ON` so the local build script (which builds the now-gtest-gated targets unconditionally) still finds them.
ghstack-source-id: 399812941
@exported-using-ghexport

Differential Revision: [D110536636](https://our.internmc.facebook.com/intern/diff/D110536636/)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. meta-exported

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants