feat: add --max-suite-retries option to cap total reruns across suite by Borda · Pull Request #332 · pytest-dev/pytest-rerunfailures

Borda · 2026-06-03T05:41:24Z

This pull request introduces a new feature to cap the total number of test reruns across the entire suite using the --max-suite-retries option. This allows users to limit overall reruns, regardless of per-test retry settings, to better control resource usage in large or flaky test suites. The implementation includes command-line interface changes, core logic updates, and comprehensive tests.

New Feature: Suite-wide rerun cap

Added a --max-suite-retries option to the CLI, allowing users to specify a maximum total number of reruns across the entire test suite. Once this cap is reached, no further reruns are performed, regardless of individual test retry settings. [1] [2] [3]

Implementation: Core logic and state management

Extended the StatusDB and related classes to track and increment a suite-wide rerun counter in a thread-safe manner, supporting both in-memory and socket-based backends. [1] [2] [3] [4]
Updated the test execution protocol to check the suite-wide rerun count before performing a rerun, ensuring the global cap is enforced during test runs.

Testing: Validation of new behavior

Added multiple tests to verify that the suite-wide rerun cap works as intended, including edge cases such as zero cap, cap not reached, and interaction with passing tests.

New CLI option --max-suite-retries (int, default None = no limit); once the suite-wide rerun count hits the cap, failing tests are logged as final failures without further retry
StatusDB gains thread-safe increment_suite_reruns() / get_suite_reruns() via threading.Lock for the single-process path
ServerStatusDB overrides use the existing rerunfailures_db dict with a "__suite__" key and an atomic socket inc operation in run_connection
ClientStatusDB overrides route increment through the socket to the master
Five new tests covering: cap enforcement, pass-through when under cap, zero disables all reruns, passing tests don't consume the budget, and standalone option no-op
CHANGES.rst and README.rst updated

Fixes #298

- New CLI option `--max-suite-retries` (int, default None = no limit); once the suite-wide rerun count hits the cap, failing tests are logged as final failures without further retry - `StatusDB` gains thread-safe `increment_suite_reruns()` / `get_suite_reruns()` via `threading.Lock` for the single-process path - `ServerStatusDB` overrides use the existing `rerunfailures_db` dict with a `"__suite__"` key and an atomic socket `inc` operation in `run_connection` - `ClientStatusDB` overrides route increment through the socket to the master - Five new tests covering: cap enforcement, pass-through when under cap, zero disables all reruns, passing tests don't consume the budget, and standalone option no-op - CHANGES.rst and README.rst updated Fixes pytest-dev#298 --- Co-authored-by: Claude Code <noreply@anthropic.com>

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds a new CLI option to cap total reruns across an entire pytest session, preventing excessive reruns in large/flaky suites.

Changes:

Introduce --max-suite-retries option and enforce it during rerun scheduling.
Add test coverage for suite-wide rerun caps (including 0 and “no effect” scenarios).
Document the new option and note it in the changelog (Fixes #298).

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
`src/pytest_rerunfailures.py`	Adds the new CLI option and suite-wide rerun counting (including xdist socket DB support).
`tests/test_pytest_rerunfailures.py`	Adds functional tests validating suite-wide rerun cap behavior.
`README.rst`	Documents how to use `--max-suite-retries` and its intent.
`CHANGES.rst`	Records the new feature in the unreleased changelog.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[resolve pytest-dev#3] Review by @Copilot (PR pytest-dev#332): "This increments the suite-wide counter even when the cap is already exhausted, which can permanently overshoot the configured limit..." Challenge: evidence=VALID suggestion=VALID resolution=as-suggested - Add `try_increment_suite_reruns(max_cap)` to StatusDB, ServerStatusDB, ClientStatusDB - ServerStatusDB adds `try_inc` socket protocol (atomic check-then-increment) - Caller uses bool return instead of post-increment comparison - Counter now reflects reruns actually performed, not attempts --- Co-authored-by: Claude Code <noreply@anthropic.com> Co-authored-by: OpenAI Codex <codex@openai.com>

[resolve pytest-dev#1] Review by @Copilot (PR pytest-dev#332): "group._addoption is a private pytest API and can break across pytest versions." Challenge: evidence=VALID suggestion=VALID resolution=as-suggested --- Co-authored-by: Claude Code <noreply@anthropic.com> Co-authored-by: OpenAI Codex <codex@openai.com>

--- Co-authored-by: Claude Code <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

[resolve pytest-dev#5] Review by @Copilot (PR pytest-dev#332): "--max-suite-retries currently accepts negative integers (because `type=int`)..." Challenge: evidence=VALID suggestion=VALID resolution=as-suggested --- Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> Co-authored-by: OpenAI Codex <codex@openai.com>

[resolve pytest-dev#6] Review by @Copilot (PR pytest-dev#332): "get_suite_reruns() reads _suite_rerun_count without the lock..." Challenge: evidence=VALID suggestion=VALID resolution=as-suggested --- Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> Co-authored-by: OpenAI Codex <codex@openai.com>

[resolve pytest-dev#7] Review by @Copilot (PR pytest-dev#332): "This test manually inspects parseoutcomes() while the surrounding..." Challenge: evidence=VALID suggestion=REJECT resolution=self-resolved --- Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> Co-authored-by: OpenAI Codex <codex@openai.com>

--- Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

        if config.option.reruns != 0:
            if config.option.usepdb:  # a core option
                raise pytest.UsageError("--reruns incompatible with --pdb")
+        if (
+            config.option.max_suite_retries is not None
+            and config.option.max_suite_retries < 0
+        ):
+            raise pytest.UsageError("--max-suite-retries must be >= 0")


icemac · 2026-06-26T06:45:48Z

+    group.addoption(
+        "--max-suite-retries",
+        action="store",
+        dest="max_suite_retries",
+        type=int,
+        default=None,
+        help="Maximum total number of reruns across the entire test suite. "
+        "Once this limit is reached, no further reruns will occur.",
+    )


I'd also prefer to call it --max-suite-reruns to make it consistent. (The repository itself also contains rerun in its name.

+                max_suite_reruns = item.session.config.option.max_suite_retries
+                if max_suite_reruns is not None:
+                    if not db.try_increment_suite_reruns(max_suite_reruns):
+                        # suite-wide limit exhausted — log as final failure
+                        item.ihook.pytest_runtest_logreport(report=report)
+                        continue


icemac

I think we are a step forward now, there are some requested changes, and please also look at the open Copilot conversations.

icemac · 2026-06-26T06:45:48Z

+    group.addoption(
+        "--max-suite-retries",
+        action="store",
+        dest="max_suite_retries",
+        type=int,
+        default=None,
+        help="Maximum total number of reruns across the entire test suite. "
+        "Once this limit is reached, no further reruns will occur.",
+    )


I'd also prefer to call it --max-suite-reruns to make it consistent. (The repository itself also contains rerun in its name.

Copilot AI review requested due to automatic review settings June 3, 2026 05:41

Copilot AI reviewed Jun 3, 2026

View reviewed changes

Comment thread src/pytest_rerunfailures.py Outdated

Comment thread src/pytest_rerunfailures.py

Comment thread src/pytest_rerunfailures.py

Comment thread src/pytest_rerunfailures.py Outdated

icemac and others added 4 commits June 3, 2026 08:54

Merge branch 'master' into feat/298

d7e415f

lint: auto-fix violations after resolve cycle

c48a198

--- Co-authored-by: Claude Code <noreply@anthropic.com>

Borda requested a review from Copilot June 4, 2026 13:29

Copilot AI reviewed Jun 4, 2026

View reviewed changes

Comment thread src/pytest_rerunfailures.py

Comment thread src/pytest_rerunfailures.py Outdated

Comment thread tests/test_pytest_rerunfailures.py Outdated

Borda and others added 4 commits June 9, 2026 19:57

lint: type annotations + mypy config for suite-retries additions

abdf2ea

--- Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

Borda requested a review from Copilot June 9, 2026 18:44

Copilot AI reviewed Jun 9, 2026

View reviewed changes

icemac requested changes Jun 26, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: add --max-suite-retries option to cap total reruns across suite#332

feat: add --max-suite-retries option to cap total reruns across suite#332
Borda wants to merge 9 commits into
pytest-dev:masterfrom
Borda:feat/298

Borda commented Jun 3, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

icemac Jun 26, 2026

Uh oh!

icemac left a comment

Uh oh!

icemac Jun 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Borda commented Jun 3, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

icemac Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

icemac left a comment

Choose a reason for hiding this comment

Uh oh!

icemac Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants