Description
The visual test report uploaded as a CI artifact determines test status from which files exist on disk, not from the test run. A test that never ran is indistinguishable from a test that failed.
On a run, vitest reported:
Test Files 1 failed | 69 passed | 4 skipped (74)
Tests 1 failed | 2123 passed | 12 skipped | 213 todo (2349)
The report generated from that same run says 73 failed / 249 passed out of 322 tests, and 120 of 484 screenshots failing. Exactly one of those 120 screenshots has a diff image.
Cause
visual-report.js builds its test list by walking test/unit/visual/screenshots/**/metadata.json, then sets status with:
passed: hasExpected && hasActual && !hasDiff
So "no actual image on disk" renders as FAIL. Three situations produced that in this run:
| Situation |
Count |
Test's project was not invoked. ci-test.yml runs --project=unit-tests, which excludes unit-tests-webgpu |
71 |
Test is skipped in source (skip: true) |
1 |
The second screenshot of the one genuinely failing test, since visualTest.js throws inside the comparison loop on the first mismatch and never captures the rest |
1 |
Only the third belongs to a test that actually failed, and it is rendered as a second independent failure next to the real one.
What could be done is to have the report read the run's results rather than infer them. vitest can emit them alongside the console reporter:
npm test -- --project=unit-tests --reporter=default --reporter=json --outputFile.json=<path>
assertionResults[] gives status (passed / failed / skipped) and ancestorTitles, which maps onto the screenshot directory name using the same / to %2F escaping visualTest.js applies. Tests excluded by --project are absent from the JSON entirely.
That gives the report four states instead of two: passed, failed, skipped, not run. Skipped and not-run come out of the failure count and the summary percentages.
Description
The visual test report uploaded as a CI artifact determines test status from which files exist on disk, not from the test run. A test that never ran is indistinguishable from a test that failed.
On a run, vitest reported:
Test Files 1 failed | 69 passed | 4 skipped (74)
Tests 1 failed | 2123 passed | 12 skipped | 213 todo (2349)
The report generated from that same run says 73 failed / 249 passed out of 322 tests, and 120 of 484 screenshots failing. Exactly one of those 120 screenshots has a diff image.
Cause
visual-report.jsbuilds its test list by walkingtest/unit/visual/screenshots/**/metadata.json, then sets status with:So "no actual image on disk" renders as FAIL. Three situations produced that in this run:
ci-test.ymlruns--project=unit-tests, which excludesunit-tests-webgpuskip: true)visualTest.jsthrows inside the comparison loop on the first mismatch and never captures the restOnly the third belongs to a test that actually failed, and it is rendered as a second independent failure next to the real one.
What could be done is to have the report read the run's results rather than infer them. vitest can emit them alongside the console reporter:
assertionResults[]givesstatus(passed/failed/skipped) andancestorTitles, which maps onto the screenshot directory name using the same/to%2FescapingvisualTest.jsapplies. Tests excluded by--projectare absent from the JSON entirely.That gives the report four states instead of two: passed, failed, skipped, not run. Skipped and not-run come out of the failure count and the summary percentages.