[REQUEST] Support parallel testing

[mrleeio]

Description

Splitting test files into multiple processes on continuous integration platforms has become common to reduce run time.

For example, on GitHub Actions, you might create a workflow which utilizes a matrix strategy. You can read a blog post on the subject here.

The compare mode for next_rails is too simplistic to account for this type of setup. Each process will create a different shitlist file which fails the diff comparison.

Possible Implementation

Create a new comparison mode that checks file content.

Committed file:

  "./spec/file_one.rb": [
    "DEPRECATION WARNING: Using `return`, `break` or `throw` to exit a transaction block is deprecated without replacement. ...",
    "DEPRECATION WARNING: Using `return`, `break` or `throw` to exit a transaction block is deprecated without replacement. ..."
  ],
  "./spec/file_two.rb": [
    "DEPRECATION WARNING: Rendering actions with '.' in the name is deprecated: ...",
    "DEPRECATION WARNING: Rendering actions with '.' in the name is deprecated: ..."
  ]

Matrix Job One:

  "./spec/file_one.rb": [
    "DEPRECATION WARNING: Using `return`, `break` or `throw` to exit a transaction block is deprecated without replacement. ...",
    "DEPRECATION WARNING: Using `return`, `break` or `throw` to exit a transaction block is deprecated without replacement. ..."
  ]

Matrix Job Two:

  "./spec/file_two.rb": [
    "DEPRECATION WARNING: Rendering actions with '.' in the name is deprecated: ...",
    "DEPRECATION WARNING: Rendering actions with '.' in the name is deprecated: ..."
  ]

Both Matrix jobs fail when comparing using a diff. However, it would pass if we searched for that content in the committed file.

Resources:

• Matrix strategy
• Parallel GitHub Actions for RSpec

I will abide by the code of conduct

[etagwerker]

@mrleeio Thanks for submitting this issue!

I'm curious, how are you running the diff command? Is it running in CI or in your dev environment?

For a similar problem (parallelized jobs in CI) with SimpleCov data, we usually recommend merging SimpleCov files from all the different artifacts generated in CI and then using them.

I'm not sure what's the best solution, but maybe doing a merge of the JSON files might be necessary...

[mrleeio]

@etagwerker sorry I may have been unclear. I only use the default comparison mode.

DEPRECATION_TRACKER=compare rspec

It will always fail when this is run on separate processes being compared against a single shitlist file. Spec distribution among processes is completely random.

This command works wonderfully locally or when specs are run within a single process.

[etagwerker]

This command works wonderfully locally or when specs are run within a single process.

I think this library could support parallel testing using your local environment, but I think it'd be too complicated to make it work in CI.

@JuanVqz just implemented a "shard merger" for deprecations that happen in separate jobs, but even that won't solve this problem.

I might be missing something, so I won't close this issue until I hear back from @JuanVqz, @arielj, or @mrleeio

[JuanVqz]

@etagwerker I feel like the parallel support we recently added in #177 can cover this scenario if you can follow this in a CI workflow.

Warning

Make sure you pass the the correct ENV["TEST_ENV_NUMBER"] at node_index

Step 1: Run Save mode

  DeprecationTracker.track_rspec(
    mode: ENV["DEPRECATION_TRACKER"],
    node_index: ENV["TEST_ENV_NUMBER"] # :save
  )

Step 2: After all processes finish, merge the shards into one file:

DeprecationTracker.merge_shards("spec/support/deprecation_warning.shitlist.json")

or the command line

deprecations merge

Step 3: Compare mode

Switch to compare mode:

  DeprecationTracker.track_rspec(
    shitlist_path: "spec/support/deprecation_warning.shitlist.json",
    mode: ENV["DEPRECATION_TRACKER"] # :compare
  )

I might be wrong about this fully solving your use case since we haven't tested it specifically with the parallel_tests gem, but the mechanism should work.

@mrleeio if you get a chance, please test it out and let us know how it goes. We're interested in knowing whether this works well for your setup.