🚀 Feature Request
Make the use of --last-failed a little nicer in CI by allowing the location to be set.
While not designed for use in CI, the last-failed flag can actually be pretty useful when retrying a failed shard, as it's one of the only post-shard filters. Meaning that the last-run file is already representative of what should be run on the current shard and shouldn't be further sharded.
The current location requires the CI configuration to be aware of the first project's output directory.
Example
Assume:
GitHub Actions matrix CI setup where there is a per-shard per pipeline cache setup for the last-run file.
- Each shard tries to grab the last-run file from cache.
- If the file exists, the ci setup will append the --last-failed command to the playwright test run.
- On error the last-run file is saved to cache.
Now despite there being multiple retries, a network hickup resulted in one test on one shard failing the test run, blocking a software release. We can now rerun just that failed job to only target the failed tests on that shard.
This already works today, but requires the CI setup to know the first project outputDir, which even if known, isn't very clear why it's in the ci config if you are actually running multiple projects with different outputDirs.
Motivation
Make it a bit simpler and cleaner to cache/restore the last-run file per-shard in CI. And to also extract the last-run.json file from the test-results folder, so it doesn't get removed on startup. Allowing a bit more flexibility, like being more easily able to keep it around on CI retry.
🚀 Feature Request
Make the use of --last-failed a little nicer in CI by allowing the location to be set.
While not designed for use in CI, the last-failed flag can actually be pretty useful when retrying a failed shard, as it's one of the only post-shard filters. Meaning that the last-run file is already representative of what should be run on the current shard and shouldn't be further sharded.
The current location requires the CI configuration to be aware of the first project's output directory.
Example
Assume:
GitHub Actions matrix CI setup where there is a per-shard per pipeline cache setup for the last-run file.
Now despite there being multiple retries, a network hickup resulted in one test on one shard failing the test run, blocking a software release. We can now rerun just that failed job to only target the failed tests on that shard.
This already works today, but requires the CI setup to know the first project outputDir, which even if known, isn't very clear why it's in the ci config if you are actually running multiple projects with different outputDirs.
Motivation
Make it a bit simpler and cleaner to cache/restore the last-run file per-shard in CI. And to also extract the last-run.json file from the test-results folder, so it doesn't get removed on startup. Allowing a bit more flexibility, like being more easily able to keep it around on CI retry.