Set timeout for both test function only and test function + all fixture setup and teardown

[eivindjahren]

It would be nice if it was possible to set a separate timeout for calling the function instead of just setting that the timeout applies only to the function. Currently I can set timeout_func_only = true in pyproject.toml, but that means that all timeouts apply only to the test function.

Concretely, The problem is that I have two kind of timeouts.

I would like to enforce that a selection of tests are fast when fixtures that can be cached are ignored. I currently run those like this:

pytest -m "not slow" tests/ --timeout=10

with timeout_func_only = true in pyproject.toml.

This is to maintain a set (the majority in our case, more than 2k), can be ran quickly (takes less than 40s with caching and pytest-xdist). Each of those tests should take less than 0.1s so a generous timeout of 10s seems appropriate. Only setting a session timeout for this run is not as reliable as it can vary widely because of pytest-xdist.

However, when running all tests in a CI runner, I need to set an appropriate large timeout (roughly 300s) for each test in order to avoid that a hanging test racks up billable minutes. A session timeout works, but needs to be set to an even larger timeout in order to avoid spurious fails due to variation. For this use case I would like to have timeout_func_only = false as slow fixtures matters for billable minutes.

[flub]

So you want:

• 60s timeout on all setup, test and teardown
• 5s timeout on the test itself

Is there a genuine problem you're trying to solve or something you consider to be nice-to-have?

If there's a genuine problem, let's understand what it is and figure out the right solution.

Otherwise I'm inclined to say no, even if you'd provide a PR. Because every bit of configuration makes things more complex.

[eivindjahren]

@flub The problem is that I have two kind of timeouts.

1. I would like to enforce that a selection of tests are fast when fixtures that can be cached are ignored. This is to maintain a set (the majority in our case, more than 2k), can be ran quickly (takes less than 40s with caching and pytest-xdist). Each of those tests should take less than 0.1s so a generous timeout of 10s seems appropriate. Setting a session timeout for this run only is not as nice as can vary widely because of pytest-xdist.
2. I need to set an appropriate large timeout (roughly 300s) for each test in order to avoid that a hanging test racks up CI minutes. A session timeout works, but needs to be set to an even larger timeout in order to avoid spurious fails due to variation.

Since func_only can only be set per-function in code or in pyproject.toml, I cannot achieve both of these without editing the file, which is not so nice for automation. My issue could be resolved by being able to set this option via cli.

[flub]

My issue could be resolved by being able to set this option via cli.

You don't mention the cli in your proposal?

[eivindjahren]

My issue could be resolved by being able to set this option via cli.

You don't mention the cli in your proposal?

Yes, my mistake. Should I reword the proposal to include cli? Or should I create a new proposal for --timeout-func-only?

[nicoddemus]

Perhaps this would be better defined as a configuration instead? CLI options are used usually for things you want to use sporadically from the command-line, while configuration options are for things that are innate of the current test suite, something that should always be set when running the tests.

[eivindjahren]

Perhaps this would be better defined as a configuration instead? CLI options are used usually for things you want to use sporadically from the command-line, while configuration options are for things that are innate of the current test suite, something that should always be set when running the tests.

The reason to have a cli-option here is that there are several ways of running the test-suite by choosing tests with a certain marker. It is quite common to have a marker @pytest.mark.slow to avoid running slow tests in certain environments or while working by using pytest -m "not slow" tests/.

[nicoddemus]

Ahh OK it seems I misunderstood the issue, sorry for the noise.

[flub]

You can keep using this issue to describe what you need. My current understanding is that you want two timeouts on each test:

1. The test function only.
2. The test function + all fixture setup and teardown.

And then it probably makes sense to have all the usual ways of setting those timers that pytest-timeout supports. Does that match?

If so I'd encourage to update the title and description of the issue so it's a bit less confusing. Just make clear that the description has been edited from the original description.

[eivindjahren]

@flub I have updated the description and and title. Tried my best to capture what you described :) .