From b619cb26ce2cf61f3d73b1a5e09e5e675b48aee9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mar=C3=ADa=20de=20Ant=C3=B3n?= Date: Thu, 4 Jun 2026 17:21:48 +0200 Subject: [PATCH 1/2] docs: Checkly-applied Playwright defaults and clarify globalTimeout support - Add table of Checkly defaults (workers, retries, video, screenshot) and enforced settings (trace, reporter, user-agent) to configuration page - Replace trace config examples with globalTimeout and max-failures examples - Rewrite environment-variables page to remove trace advice and show workers/baseURL as the conditional config example - Update timeouts page to reflect globalTimeout as user-configurable --- constructs/playwright-check.mdx | 6 +- .../playwright-checks/configuration.mdx | 57 ++++++++++---- .../environment-variables.mdx | 77 +++++++++++-------- .../playwright-checks/timeouts.mdx | 39 +++++----- 4 files changed, 112 insertions(+), 67 deletions(-) diff --git a/constructs/playwright-check.mdx b/constructs/playwright-check.mdx index cc139269..9e75322a 100644 --- a/constructs/playwright-check.mdx +++ b/constructs/playwright-check.mdx @@ -45,7 +45,7 @@ new PlaywrightCheck("critical-e2e-monitor", { playwrightConfigPath: "./playwright.config.ts", engine: Engine.node("24"), // Optional: override the auto-detected engine. installCommand: "npm ci", - testCommand: "npx playwright test --retries=2", + testCommand: "npx playwright test --max-failures=5", pwProjects: ["chromium"], pwTags: ["e2e"], }) @@ -145,6 +145,10 @@ new PlaywrightCheck("critical-e2e-monitor", { If you configure the `testCommand` property and `pwTags` or `pwProjects` the options will be merged and all configurations applied to `npx playwright test`. + +Checkly applies these Playwright defaults when you do not set them yourself: `workers: 4`, `retries: 2`, `use.video: "on-first-retry"`, and `use.screenshot: "on-first-retry"`. Checkly also records traces on every run, uses the Checkly reporter, and appends a Checkly user-agent suffix. Checkly supports Playwright's `globalTimeout`; the timer starts when `npx playwright test` begins. + + diff --git a/detect/synthetic-monitoring/playwright-checks/configuration.mdx b/detect/synthetic-monitoring/playwright-checks/configuration.mdx index 4fd124d5..b080dc1a 100644 --- a/detect/synthetic-monitoring/playwright-checks/configuration.mdx +++ b/detect/synthetic-monitoring/playwright-checks/configuration.mdx @@ -7,7 +7,7 @@ sidebarTitle: 'Configuration' Use the `checkly.config.ts/js` file to define your Playwright Check Suite. Each Playwright Check Suite can be connected to references in your `playwright.config.ts/js` file. -**A Playwright Check Suite can run up to 30 minutes**. [Please contact us in the Checkly Web App](https://app.checklyhq.com/?support=true) or get in touch with your account executive if you're interested in longer runs. +**The Playwright test execution stage can run for up to 30 minutes**. Use Playwright's `globalTimeout` to set a custom suite-level limit. [Contact us in the Checkly Web App](https://app.checklyhq.com/?support=true) or get in touch with your account executive if you need longer runs. ## Playwright Check Suite definition @@ -103,14 +103,39 @@ A Playwright Check Suite inherits multiple properties from the abstract `Check` Check out [the reference documentation](/constructs/project#param-checks-playwright-checks) for more information on these settings. -**Checks' Retry strategy is not applicable for Playwright checks.** Playwright includes its own retry features that can be set up directly in your `playwright.config.ts/js` file with the [`retries`](https://playwright.dev/docs/test-retries) option. This allows for more detailed management of test retries within Playwright, when your check runs. +**Checkly retry strategies do not apply to Playwright Check Suites.** Use Playwright's [`retries`](https://playwright.dev/docs/test-retries) option in your `playwright.config.ts/js` file or `testCommand` to control test retries. -Additionally, Playwright Check Suites provide specific configuration for your Playwright monitoring. +### Checkly-applied Playwright settings + +When Checkly runs a Playwright Check Suite, it applies Playwright settings that produce useful monitoring results and artifacts. + +Checkly applies these defaults only when you do not set the option yourself: + +| Playwright option | Checkly default | +|-------------------|-----------------| +| `workers` | `4` | +| `retries` | `2` | +| `use.video` | `on-first-retry` | +| `use.screenshot` | `on-first-retry` | + +Your Playwright config or `testCommand` value takes precedence for these options. + +Checkly enforces these options for every run. User configuration does not override them: + +| Playwright setting | Checkly behavior | +|--------------------|------------------| +| `use.trace` | `on` for every run | +| Reporter | [Checkly reporter](/detect/testing/playwright-reporter) | +| User agent | Checkly appends a [user-agent suffix](/platform/allowlisting-traffic#default-checkly-user-agent) | + +Checkly supports Playwright's [`globalTimeout`](https://playwright.dev/docs/api/class-testconfig#test-config-global-timeout). Use it to control the maximum duration of the Playwright suite after `npx playwright test` starts. + +Playwright Check Suites also provide specific configuration for your Playwright monitoring. * `installCommand:` Override the command to install dependencies. `npm install --dev` is used by default. -* `testCommand:` Override the command to test. `npx playwright test` is used by default with the tags, projects, and config file options your check specifies. +* `testCommand:` Override the command that runs tests. `npx playwright test` is used by default with the tags, projects, and config file options your check specifies. * `group:` The group construct this check belongs to. @@ -132,22 +157,28 @@ installCommand: 'yarn install --frozen-lockfile' installCommand: 'npm install --prefix ./e2e' ``` -**Enable Playwright tracing to debug failures** +**Set a lower test timeout:** ```typescript -testCommand: 'npx playwright test --trace=on' +testCommand: 'npx playwright test --timeout=60000' ``` -**Set a specific test timeout:** +**Set a global suite timeout:** ```typescript -testCommand: 'npx playwright test --timeout=60000' +testCommand: 'npx playwright test --global-timeout=120000' +``` + +**Stop early after failures:** + +```typescript +testCommand: 'npx playwright test --max-failures=5' ``` **Combine multiple test flags:** ```typescript -testCommand: 'npx playwright test --trace=retain-on-failure --max-failures=5' +testCommand: 'npx playwright test --timeout=60000 --max-failures=5' ``` ### Example Checkly config @@ -169,7 +200,7 @@ export default defineConfig({ pwProjects: ['chromium', 'firefox', 'webkit'], pwTags: '@e2e', installCommand: 'npm install --dev', - testCommand: 'npx playwright test --trace=on', + testCommand: 'npx playwright test --max-failures=5', activated: true, muted: false, group: productionGroup, @@ -318,7 +349,7 @@ If tests fail due to authentication issues: - Verify environment variables are set correctly in Checkly. You can verify your environment variables using `npx checkly env ls`, or looking at your Global or Check's environment variables in the Checkly Webapp to ensure any `process.env.VARIABLE_NAME` call from your test is defined. - For persistent authentication, use Playwright's [`storageState`](https://playwright.dev/docs/auth#reuse-authentication-state) feature. -**webServer configuration requires additional files** +#### webServer configuration requires additional files When using `webServer` in your Playwright config, you need to include the server files in your check configuration: @@ -326,9 +357,9 @@ When using `webServer` in your Playwright config, you need to include the server - Include all files needed to start the server (e.g., server scripts, config files) - Alternatively, skip `webServer` on Checkly and use your deployed URL: `...(process.env.CHECKLY ? {} : { webServer: { ... } })` -**No tests found** +#### No tests found -If Playwright couldn't find any tests to run, this usually means a mismatch between your config and your test files: +If Playwright does not find any tests to run, this usually means a mismatch between your config and your test files: - Verify your `--grep` pattern matches actual test tags or projects - Verify the `testDir` path in `playwright.config.ts` matches the actual directory containing your test files diff --git a/detect/synthetic-monitoring/playwright-checks/environment-variables.mdx b/detect/synthetic-monitoring/playwright-checks/environment-variables.mdx index 83da5da5..a2b56800 100644 --- a/detect/synthetic-monitoring/playwright-checks/environment-variables.mdx +++ b/detect/synthetic-monitoring/playwright-checks/environment-variables.mdx @@ -6,82 +6,92 @@ tags: ["synthetic-monitoring", "playwright-checks", "environment-variables"] --- - For information on creating and managing environment variables and secrets (including the difference between variables and secrets, inheritance, and where to set them), see the [Environment Variables documentation](/platform/variables). + To create and manage environment variables and secrets, including inheritance and where to set them, see the [Environment Variables documentation](/platform/variables). -## Built-in variables available in Playwright Check Suites +## Built-in variables in Playwright Check Suites Checkly sets the following environment variables on every Playwright Check Suite run: | Variable | Description | |----------|-------------| -| `CHECKLY` | Set to `1` for all check runs executed by Checkly.
Useful to distinguish Checkly runs from local runs. | -| `CHECKLY_RUN_SOURCE` | Indicates how the check was triggered.
Useful to customize the nature of your test depending on how it is run.
See [CHECKLY_RUN_SOURCE values](#checkly-run-source-values) below. | -| `CI` | Set to `1` for CLI runs (`npx checkly test` or `npx checkly trigger`) and [deployment](/integrations/ci-cd/github/deployments)-triggered checks. | +| `CHECKLY` | Checkly sets this to `1` for every Playwright Check Suite run. Use it to distinguish Checkly runs from local runs. | +| `CHECKLY_RUN_SOURCE` | Identifies what triggered the check run. Use it to adjust test behavior for schedules, manual triggers, deployments, or CLI runs. See [`CHECKLY_RUN_SOURCE` values](#checkly-run-source-values). | +| `CI` | Checkly sets this to `1` for CLI runs (`npx checkly test` or `npx checkly trigger`) and checks started by [deployments](/integrations/ci-cd/github/deployments). | | `ACCOUNT_ID` | The UUID of the Checkly account. | | `CHECK_NAME` | The name of the check. | | `CHECKLY_CHECK_ID` | The UUID of the check. | -| `CHECKLY_REGION` | The region where the check was executed. | +| `CHECKLY_REGION` | The region where Checkly ran the check. | ### `CHECKLY_RUN_SOURCE` values | Value | Description | |-------|-------------| -| `CLI_DEPLOY` | Used for the first run of Checks deployed using `npx checkly deploy`. | -| `DEPLOYMENT` | Used for Check runs triggered as part of a [CI/CD deployment](/integrations/ci-cd/github/deployments). | -| `GROUP_RUN_ALL` | Used for Check runs triggered from a Group's edit screen, by clicking the "Run all checks" button. | -| `SCHEDULE_NOW` | Used for Check runs triggered manually by clicking "Schedule now" in the webapp. | -| `SCHEDULER` | Used for Check runs triggered as part of their regular schedule. | -| `TEST_NO_RECORD` | Used for Check runs triggered by the `npx checkly test` CLI command. | -| `TEST_RECORD` | Used for Check runs triggered by the `npx checkly test --record` or `npx checkly pw-test` CLI commands. | -| `TRIGGER_API` | Used for Check runs triggered by the API. | -| `TRIGGER_NO_RECORD` | Used for Check runs triggered by the `npx checkly trigger` CLI command. | -| `TRIGGER_RECORD` | Used for Check runs triggered by the `npx checkly trigger --record` CLI command. | +| `CLI_DEPLOY` | The first run after `npx checkly deploy`. | +| `DEPLOYMENT` | A [CI/CD deployment](/integrations/ci-cd/github/deployments) started the run. | +| `GROUP_RUN_ALL` | The "Run all checks" action on the group's edit screen started the run. | +| `SCHEDULE_NOW` | The "Schedule now" action in the web app started the run. | +| `SCHEDULER` | The regular check schedule started the run. | +| `TEST_NO_RECORD` | `npx checkly test` started the run. | +| `TEST_RECORD` | `npx checkly test --record` or `npx checkly pw-test` started the run. | +| `TRIGGER_API` | The API started the run. | +| `TRIGGER_NO_RECORD` | `npx checkly trigger` started the run. | +| `TRIGGER_RECORD` | `npx checkly trigger --record` started the run. | ## Use built-in variables -Common ways to use Checkly's built-in environment variables: - +Use Checkly's built-in environment variables to tune test behavior for different run contexts. ### Adjust behavior based on trigger type -Set different behavior for scheduled runs versus manual triggers: +Use different timeouts for scheduled runs and manual triggers: ```typescript highlight={4} import { test } from '@playwright/test'; test('API health check', async ({ request }) => { + // Scheduled monitors should fail fast. Manual runs can wait longer for debugging. const isScheduledRun = process.env.CHECKLY_RUN_SOURCE === 'SCHEDULER'; - - // Use shorter timeout for scheduled runs const timeout = isScheduledRun ? 5000 : 30000; const response = await request.get('https://api.example.com/health', { - timeout + timeout, }); // ... assertions }); ``` -### Configure Playwright based on environment - -Adjust your Playwright configuration to always record traces on Checkly. +### Configure Playwright by environment -In your `playwright.config.ts`, use the CHECKLY variable to identify it's from Checkly. +Use the built-in [`CHECKLY` environment variable](#built-in-variables-in-playwright-check-suites) in your `playwright.config.ts` to use different settings for Checkly runs and local development. -```typescript playwright.config.ts highlight={7} +```typescript playwright.config.ts highlight={5-9} import { defineConfig } from '@playwright/test'; export default defineConfig({ - + // Run one worker in Checkly when tests share state or accounts. + workers: process.env.CHECKLY === '1' ? 1 : undefined, use: { - // Always trace in Checkly to compare successful and failed runs, only on retry elsewhere - trace: process.env.CHECKLY === '1' ? 'on' : 'on-first-retry', + // Checkly runs against a public URL. Local runs use the dev server. + baseURL: process.env.CHECKLY === '1' + ? 'https://www.example.com' + : 'http://localhost:3000', }, }); ``` + +Playwright Check Suites record traces for successful and failed runs. Checkly uses these traces to show dedicated Network and Browser console sections, so you can compare requests, logs, and errors when you troubleshoot. + + +You can also adjust these settings for Checkly runs: + +- `maxFailures`: stop early during monitoring runs, for example after the first failure. +- `timeout`: give tests more or less time in Checkly than during local development. +- `expect.timeout`: tune assertion timeouts for production or staging environments. +- `retries`: override Checkly's default retry count if your suite needs a different policy. + ### Access region information Use region information for debugging or region-specific behavior: @@ -92,13 +102,12 @@ import { test } from '@playwright/test'; test('geo-specific content test', async ({ page }) => { await page.goto('https://example.com'); - // Log region for debugging + // Include the region in logs to compare results across locations. console.log(`Running in region: ${process.env.CHECKLY_REGION}`); - // Region-specific assertions + // Add assertions only for regions where the content should appear. if (process.env.CHECKLY_REGION?.startsWith('eu-')) { - // Expect EU-specific content, like a cookie banner + // Example: check for an EU-specific cookie banner. } }); ``` - diff --git a/detect/synthetic-monitoring/playwright-checks/timeouts.mdx b/detect/synthetic-monitoring/playwright-checks/timeouts.mdx index aae34449..4292d3fa 100644 --- a/detect/synthetic-monitoring/playwright-checks/timeouts.mdx +++ b/detect/synthetic-monitoring/playwright-checks/timeouts.mdx @@ -11,7 +11,7 @@ Use this guide to configure your checks and troubleshoot timeout failures. | Timeout stage | Default timeout | Configurable | |---------------|-----------------|--------------| -| Global Playwright Check Suite timeout | 30 minutes | No | +| Playwright `globalTimeout` | Inherits from your config | Yes, up to 30 minutes | | Dependency installation | 5 minutes | No | | Playwright test timeout | Inherits from your config | Yes | | Playwright navigation timeout | Inherits from your config | Yes | @@ -20,38 +20,39 @@ Use this guide to configure your checks and troubleshoot timeout failures. ## Playwright Check Suite execution stages -When Checkly runs your Playwright Check Suite, it has a global timeout of 30 minutes, while going through these stages: +When Checkly runs your Playwright Check Suite, it goes through these stages: ### 1. Pre-processing Checkly prepares your execution environment: -1. **Playwright and browsers**: Load the right playwright version with its corresponding browsers -2. **Dependencies cache download**: Downloads the dependencies cache if it exists -3. **Dependency installation**: If the dependencies cache doesn't exist, Checkly runs the install command (default: `npm install`, `pnpm install` or `yarn install`) +1. **Playwright and browsers**: Checkly loads the matching Playwright version and browser binaries +2. **Dependency cache download**: Checkly downloads the dependency cache if it exists +3. **Dependency installation**: If the dependency cache does not exist, Checkly runs the install command (default: `npm install`, `pnpm install`, or `yarn install`) - This stage has a **5-minute timeout** - Use `installCommand` in your [check suite configuration](/detect/synthetic-monitoring/playwright-checks/configuration#customize-install-and-test-commands) to run a custom dependency installation command. This can help prevent timeouts during installation. For example `installCommand: 'npm install --ignore-scripts'` to skip running postinstall and other lifecycle scripts -4. **Creates and uploads new dependencies cache**: This cache will be used in consecutive runs in your account with the same dependencies ### 2. Test execution Checkly runs your tests using `npx playwright test`: -- The timeouts of this stage are set through the timeouts in your Playwright tests and configuration. -- To set a global playwright test timeout in Checkly, use a custom test command, `testCommand` , in your [check suite configuration](/detect/synthetic-monitoring/playwright-checks/configuration#customize-install-and-test-commands) - - Use `testCommand: npx playwright test --global-timeout=` to customize the timeout of this step +- Your Playwright tests and configuration set the timeouts for this stage. +- To limit the whole Playwright suite, use Playwright's native [`globalTimeout`](https://playwright.dev/docs/api/class-testconfig#test-config-global-timeout) option. The timer starts when `npx playwright test` begins. + - In `playwright.config.ts`, set `globalTimeout: 120_000` + - In `testCommand`, use Playwright's [`--global-timeout`](https://playwright.dev/docs/test-cli#all-options) flag: `testCommand: 'npx playwright test --global-timeout=120000'` + - If you set `globalTimeout` in both places, the `testCommand` flag takes precedence, matching Playwright's behavior ### 3. Post-processing After tests complete, Checkly: -1. **Uploads artifacts**: Screenshots, videos, traces -2. **Scrubs secrets**: Removes secrets declared in your Checkly Environment Secrets from logs and artifacts - +1. **Uploads artifacts**: Checkly uploads screenshots, videos, and traces +2. **Scrubs secrets**: Checkly removes secrets declared in your Checkly Environment Secrets from logs and artifacts +3. **Uploads the dependency cache after successful runs**: When a check run passes, Checkly uploads the dependency cache and creates a custom runtime. Future runs can use that runtime when they come from the same repository and use the same `package.json`, lock files, and workspace dependencies. ## Playwright-specific timeouts -Checkly runs your tests with the timeout configuration from your `playwright.config.ts` and test files. Configure test timeout, navigation timeout, and action timeout according to your needs. See [Playwright's timeout documentation](https://playwright.dev/docs/test-timeouts) for detailed configuration options. +Checkly runs your tests with the timeout configuration from your `playwright.config.ts` and test files. Configure test, navigation, and action timeouts according to your needs. See [Playwright's timeout documentation](https://playwright.dev/docs/test-timeouts) for detailed configuration options. ## Troubleshooting timeout-related errors @@ -73,18 +74,18 @@ Failed to run install command See the [Custom dependencies troubleshooting guide](/detect/synthetic-monitoring/playwright-checks/custom-dependencies#troubleshooting) for more detailed solutions. -### Global timeout exceeded +### Playwright global timeout exceeded ``` -Your check took too long to complete and was automatically stopped. If this issue persists, please contact support at ... +Timed out waiting 120000ms from config.globalTimeout. ``` -**Cause**: The entire check suite execution (installation + tests + post-processing) exceeded 30 minutes. - +**Cause**: The Playwright test suite exceeded the configured `globalTimeout` after `npx playwright test` started. Playwright skipped the remaining tests and Checkly reported the check as failed. **Solution**: -- Split tests into multiple check suites using `pwTags`, `pwProjects` or a custom `testCommand`. - +- Increase `globalTimeout` in your Playwright config or `testCommand`. +- Split tests into multiple check suites using `pwTags`, `pwProjects`, or a custom `testCommand`. +- Use Playwright's per-test `timeout` option when only individual tests need more time. ### Playwright test timeout exceeded From 7dc5dddad2da4f1822ef39fa1a2a1856104406b3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mar=C3=ADa=20de=20Ant=C3=B3n?= Date: Thu, 4 Jun 2026 17:52:46 +0200 Subject: [PATCH 2/2] remove reporter, it's accurate but brings no value? --- detect/synthetic-monitoring/playwright-checks/configuration.mdx | 1 - 1 file changed, 1 deletion(-) diff --git a/detect/synthetic-monitoring/playwright-checks/configuration.mdx b/detect/synthetic-monitoring/playwright-checks/configuration.mdx index b080dc1a..775cd9fc 100644 --- a/detect/synthetic-monitoring/playwright-checks/configuration.mdx +++ b/detect/synthetic-monitoring/playwright-checks/configuration.mdx @@ -126,7 +126,6 @@ Checkly enforces these options for every run. User configuration does not overri | Playwright setting | Checkly behavior | |--------------------|------------------| | `use.trace` | `on` for every run | -| Reporter | [Checkly reporter](/detect/testing/playwright-reporter) | | User agent | Checkly appends a [user-agent suffix](/platform/allowlisting-traffic#default-checkly-user-agent) | Checkly supports Playwright's [`globalTimeout`](https://playwright.dev/docs/api/class-testconfig#test-config-global-timeout). Use it to control the maximum duration of the Playwright suite after `npx playwright test` starts.