Version
1.58.0
Steps to reproduce
- Create a minimal project with @playwright/test@1.57.0 and a playwright.config.ts that uses the JUnit reporter:
// playwright.config.ts
import { defineConfig } from '@playwright/test';
export default defineConfig({
reporter: [['junit', { outputFile: 'results.xml' }]],
retries: 0,
});
- Add a small Page Object that waits for a cookie banner — note the raw locator.waitFor(), which is the common pattern in real codebases (the banner element itself doesn't have an "expectation" to assert; you just wait for it before clicking it):
// pages/CookiesPopUp.ts
import type { Page, Locator } from '@playwright/test';
export class CookiesPopUp {
readonly banner: Locator;
readonly acceptAll: Locator;
constructor(page: Page) {
this.banner = page.locator('#cookies-banner-sdk');
this.acceptAll = page.getByRole('button', { name: /accept all/i });
}
async acceptCookies() {
await this.banner.waitFor(); // ← raw waitFor, not expect()
await this.acceptAll.click();
await this.banner.waitFor({ state: 'hidden' });
}
}
- Add a test that exercises a page where the cookie banner never renders:
// tests/some.spec.ts
import { test } from '@playwright/test';
import { CookiesPopUp } from '../pages/CookiesPopUp';
test.beforeEach(async ({ page }) => {
// Empty page — #cookie-banner-sdk never appears, so waitFor() will throw TimeoutError.
await page.setContent('<html><body>no banner here</body></html>');
await new CookiesPopUp(page).acceptCookies();
});
test('any test that depends on cookie consent', async ({ page }) => {
// Body never runs — beforeEach throws first.
});
- Run npx playwright test and inspect results.xml.
- Bump to @playwright/test@1.59.1 (or any version >= 1.58.0), re-run, and inspect results.xml again.
The same hook, throwing the same TimeoutError from the same locator.waitFor() call, ends up in two different XML elements and is counted in two different root-level attributes depending on the Playwright minor version.
This pattern (a hook that does pre-work via raw Playwright calls before the body runs) is extremely common.
Expected behavior
Either (a) default JUnit output is unchanged across the upgrade, matching the explicit "Compatibility" promise in the originating feature request #39193:
This proposal does not change default JUnit output. Existing consumers that rely on <failure> will not be affected. Only users who explicitly enable the flag will get <error> elements and errors++ counters.
…or (b) the change ships as BREAKING CHANGE: with a clear migration note in the release-notes highlights and an opt-out for downstream consumers.
In both versions the failed test should be counted in the same root-level attribute, so that consumers parsing the file with code like int(root.attrib.get("failures", 0)) continue to work.
Actual behavior
The default JUnit output silently changed in v1.58 via PR #39350 ("fix(junit): use actual error message and distinguish errors from failures"). Thrown exceptions (TimeoutError from locator.waitFor(), hook failures, navigation errors, etc.) are now serialised as and counted under the root errors="" attribute, instead of / failures="".
The change shipped as fix: (not feat:, not BREAKING CHANGE:), with no opt-in flag, no mention in the v1.58 release-notes highlights, and no discussion of compatibility impact in the PR thread — even though #39193 explicitly proposed it as opt-in to preserve compatibility.
We hit this on a real 1.57.0 → 1.59.1 bump. A CI run with 28 real TimeoutError failures produced JUnit with failures="0" errors="28", and our Slack reporter posted 🟢 SUCCESS for every broken build until we identified the cause.
Additional context
No response
Environment
System:
OS: macOS 26.5
CPU: (14) arm64 Apple M3 Max
Memory: 559.91 MB / 36.00 GB
Binaries:
Node: 22.18.0 - /Users/a616445/.nvm/versions/node/v22.18.0/bin/node
npm: 10.9.3 - /Users/a616445/.nvm/versions/node/v22.18.0/bin/npm
pnpm: 10.23.0 - /Users/a616445/.nvm/versions/node/v22.18.0/bin/pnpm
Languages:
Bash: 3.2.57 - /bin/bash
npmPackages:
@playwright/test: 1.59.1 => 1.59.1
Version
1.58.0
Steps to reproduce
The same hook, throwing the same TimeoutError from the same locator.waitFor() call, ends up in two different XML elements and is counted in two different root-level attributes depending on the Playwright minor version.
This pattern (a hook that does pre-work via raw Playwright calls before the body runs) is extremely common.
Expected behavior
Either (a) default JUnit output is unchanged across the upgrade, matching the explicit "Compatibility" promise in the originating feature request #39193:
This proposal does not change default JUnit output. Existing consumers that rely on <failure> will not be affected. Only users who explicitly enable the flag will get <error> elements and errors++ counters.…or (b) the change ships as BREAKING CHANGE: with a clear migration note in the release-notes highlights and an opt-out for downstream consumers.
In both versions the failed test should be counted in the same root-level attribute, so that consumers parsing the file with code like int(root.attrib.get("failures", 0)) continue to work.
Actual behavior
The default JUnit output silently changed in v1.58 via PR #39350 ("fix(junit): use actual error message and distinguish errors from failures"). Thrown exceptions (TimeoutError from locator.waitFor(), hook failures, navigation errors, etc.) are now serialised as and counted under the root errors="" attribute, instead of / failures="".
The change shipped as fix: (not feat:, not BREAKING CHANGE:), with no opt-in flag, no mention in the v1.58 release-notes highlights, and no discussion of compatibility impact in the PR thread — even though #39193 explicitly proposed it as opt-in to preserve compatibility.
We hit this on a real 1.57.0 → 1.59.1 bump. A CI run with 28 real TimeoutError failures produced JUnit with failures="0" errors="28", and our Slack reporter posted 🟢 SUCCESS for every broken build until we identified the cause.
Additional context
No response
Environment
System: OS: macOS 26.5 CPU: (14) arm64 Apple M3 Max Memory: 559.91 MB / 36.00 GB Binaries: Node: 22.18.0 - /Users/a616445/.nvm/versions/node/v22.18.0/bin/node npm: 10.9.3 - /Users/a616445/.nvm/versions/node/v22.18.0/bin/npm pnpm: 10.23.0 - /Users/a616445/.nvm/versions/node/v22.18.0/bin/pnpm Languages: Bash: 3.2.57 - /bin/bash npmPackages: @playwright/test: 1.59.1 => 1.59.1