Classify plugin startup import failures (#993)

[RonnyPfannschmidt]

Closes #993.

The state of #993

The issue reproduces unchanged on main. What it reports as EXIT_TESTSFAILED is not actually a chosen exit code: PytestPluginManager.import_plugin re-raises a plain ImportError, which is neither UsageError nor ConftestImportFailure, so it escapes _main entirely. CPython prints a raw traceback and exits 1. The collision with EXIT_TESTSFAILED is a coincidence.

Two consequences the issue does not mention:

• pytest.main() raises instead of returning, contradicting its documented "returns an exit code" contract.
• Example 2 in the issue (a bare raise ImportError, no args) additionally crashes pytest's own internals with IndexError: tuple index out of range from e.args[0], so the user never even sees their plugin's failure.

The rule this PR implements

The previous attempt (#7290) stalled on @bluetech's question: is a plugin failing under a top-level conftest's pytest_plugins a conftest problem or a plugin problem? This PR answers it by splitting on what actually went wrong rather than on which mechanism requested the plugin:

• The plugin cannot be found — pytest was pointed at something that is not there. That is misuse → ExitCode.USAGE_ERROR (4), the same code conftest.py import failures already return.
• The plugin was found but raised while importing — including a missing transitive dependency and a broken pytest11 entry point. That is a defect in the plugin, not misuse → ExitCode.INTERNAL_ERROR (3).

Discrimination is on ModuleNotFoundError.name matching the requested import spec (or a parent package of it), so a plugin whose own import foo fails is not misattributed to the user.

The other objection to #7290 was that routing through ConftestImportFailure hid the stack trace. Here the traceback is preserved both in the printed report and on PluginImportFailure.cause.

conftest.py import failures are deliberately unchanged and keep returning 4.

Resulting behavior

	before	after
-p nosuchplugin / pytest_plugins / PYTEST_PLUGINS naming a missing module	1 + raw traceback	4 ERROR: Error importing plugin "…"
plugin raises at import (any exception type)	1 + raw traceback	3 + formatted traceback
plugin's own dependency missing	1 + raw traceback	3
broken pytest11 entry point	1 + raw traceback	3
conftest.py fails to import	4	4 (unchanged)
pytest.main() in-process	raised ImportError	returns the exit code

Open questions for discussion

This is a draft because the classification is a judgement call and the exit code is observable API.

1. Is INTERNAL_ERROR (3) the right bucket for a broken plugin? The original issue proposed 3 for everything. wrap_session already uses 3 for exceptions during the session, including exceptions raised by plugins, so this is at least self-consistent — but 3 is documented as "internal error" and a third-party plugin blowing up is arguably not pytest's internals. The alternative is a new exit code, which is a bigger commitment.
2. Should conftest.py follow the same rule? Today any exception from a conftest is 4, so a conftest with a genuine bug in it reports as user misuse. Leaving it alone keeps the blast radius small, but it means the two paths are still not fully symmetric.
3. Is the entry-point override in scope? PytestPluginManager.load_setuptools_entrypoints is overridden so a broken installed plugin also gets 3. Without it a broken installed plugin still exits 1 with a raw traceback, which felt like an incomplete fix — but it does widen what this PR touches.
4. Changelog classification. Currently filed as bugfix. It does change observable exit codes, so it may warrant breaking instead — CI keying on 1 vs 3/4 would notice.

Tests

New TestStartupPluginImportErrors in testing/acceptance_test.py covers every entry path (-p, pytest_plugins, PYTEST_PLUGINS, entry point), the bare-ImportError regression, missing-plugin vs missing-dependency, missing submodule of an existing package, and a guard that conftest stays 4.

Six existing tests were updated from ImportError to UsageError/PluginImportFailure. test_importplugin_error_message (#375, #1998) still asserts the plugin's own frame is reachable, now through the cause.

Full suite and pre-commit (incl. mypy) pass locally.

🤖 Generated with Claude Code