[Feature]: custom reporter API for test sharding

[aslushnikov]

🚀 Feature Request

Hi team,

We'd like to implement a shard balancing logic in Flakiness.io Playwright reporter: since we have all the history for test runs across environments, we have enough data to balance shards.

We know there's --test-list API that we can use to generate a list with tests (and that's what we use for prototyping). However, this is not an ergonomic API for our clients: it requires them to change their CI configuration.

Custom Reporter API is a better integration point: there's already a full information for the upcoming test run.

Example

It would be nice if custom reporters had an early asynchronous hook that could inspect the resolved test suite and return a custom test list before execution starts.
For example, a reporter method called in the beginning of the run would do:

interface Reporter {
  /**
   * Called before the test run starts. Allows the reporter to customize
   * the list of tests that will be executed. Returns an array of test case Ids. 
   */
  selectTests?(
    config: FullConfig,
    suite: Suite
  ): undefined | Promise<undefined | string[]>;
}

Motivation

Consider this Supabase CI: if they were to try using --test-list with Flakiness.io, they'd need to amend their workflow configuration.

Additionally, if they were passing any custom arguments like --grep to their future npx playwright test run, they'd need to mirror them to the test-list generation command.