Python: [BREAKING] Graduate create_harness_agent out of experimental#7120
Python: [BREAKING] Graduate create_harness_agent out of experimental#7120westey-m wants to merge 4 commits into
Conversation
giles17
added
documentation
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Automated Code Review
Reviewers: 5 | Confidence: 90%
✓ Correctness
The PR correctly graduates create_harness_agent from experimental status while preserving targeted ExperimentalWarning emissions for still-experimental parameters. The dedup-key strategy correctly prevents double-warnings from downstream
@experimental-decoratedproviders. The stacklevel, truthiness checks, and shared mutable set usage are all correct. No correctness issues found.
✓ Security Reliability
The PR graduates
create_harness_agentfrom experimental status while preserving targeted warnings for still-experimental sub-features. The implementation follows the established_WARNED_FEATURESdedup pattern from_feature_stage.py, uses a correctstacklevel=3to point warnings at the caller's call site, and seeds the dedup set so downstream providers don't double-warn. No security vulnerabilities, resource leaks, injection risks, or unhandled failure modes were identified.
✓ Test Coverage
The PR adds good per-parameter warning tests and a graduation guard, but the central design decision highlighted by the author—the dedup-key strategy—lacks explicit test coverage. There is no test verifying that (a) repeated calls with the same experimental param only warn once, (b) multiple HARNESS params in a single call produce one combined warning naming all params, or (c) HARNESS and SHELL_TOOLING dedup keys are independent (both fire when both features are enabled). The individual tests are well-structured and assertions are meaningful.
✓ Failure Modes
The PR cleanly graduates create_harness_agent while preserving targeted experimental warnings for still-experimental features. The dedup-key mechanism correctly mirrors the existing _warn_on_feature_use pattern in _feature_stage.py, the stacklevel=3 is accurate for the fixed call path, and the warning-before-seed ordering handles all warning filter modes correctly (error → abort before seed, ignore → intentional suppression, default → seed prevents downstream double-warn). No silent failure paths or operational failure modes introduced.
✓ Design Approach
I found one design issue in the new runtime warning boundary:
create_harness_agentnow consumes the one-time experimental dedup key before some opted-in features have actually been validated or wired. That means a failed first call can suppress the warning on a later successful call in the same process.
Suggestions
- Consider deferring insertion into
_WARNED_FEATURESuntil after the corresponding feature has been successfully built/wired. Right now_warn_experimental_harness_params()adds the dedup key eagerly (python/packages/core/agent_framework/_harness/_agent.py:294-304), butcreate_harness_agent()calls it before shell validation/wiring (_agent.py:561-566before584-589), and_assemble_shell()can still raiseTypeErrorfor an invalid executor (_agent.py:241-246).
Automated review by westey-m's agents
Python Test Coverage Report •
Python Unit Test Overview
|
||||||||||||||||||||||||||||||||||||||||
There was a problem hiding this comment.
Pull request overview
Graduates the Python create_harness_agent factory out of experimental so callers can use it without opting into experimental APIs, while preserving targeted ExperimentalWarning behavior when callers enable still-experimental (or pre-release) harness features.
Changes:
- Removed the
@experimental(HARNESS)staging fromcreate_harness_agentand replaced it with runtime warnings only when specific gated parameters are used. - Implemented per-feature warning deduplication: harness-related experimental features share the
HARNESSdedup key, while shell tooling uses a separate dedup key. - Updated documentation and added tests covering the graduated default behavior and warnings for each gated parameter.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| python/packages/core/agent_framework/_harness/_agent.py | Removes the experimental decorator and adds runtime gated ExperimentalWarning emission with dedup keys. |
| python/packages/core/AGENTS.md | Documents the new experimental-feature gating behavior and dedup-key strategy. |
| python/packages/core/tests/core/test_harness_agent.py | Adds tests ensuring create_harness_agent is no longer experimental and that each gated parameter emits warnings. |
| @_requires_shell_tools | ||
| def test_create_harness_agent_shell_executor_emits_experimental_warning() -> None: | ||
| """Opting into shell tooling (pre-release) should warn and still wire the shell tool.""" | ||
| from agent_framework._feature_stage import ExperimentalWarning | ||
|
|
||
| _clear_harness_experimental_dedup() | ||
| _clear_harness_shell_dedup() | ||
| client = _FakeShellClient() | ||
| with pytest.warns(ExperimentalWarning, match="shell_executor"): | ||
| agent = create_harness_agent( | ||
| client=client, | ||
| max_context_window_tokens=128_000, | ||
| max_output_tokens=16_384, | ||
| disable_web_search=True, | ||
| disable_file_memory=True, | ||
| shell_executor=_FakeShellTool(), | ||
| ) | ||
| assert "shell_tool_instance" in agent.default_options.get("tools", []) |
There was a problem hiding this comment.
Good catch — addressed in 62cdabe: added test_create_harness_agent_shell_dedup_does_not_suppress_harness_warning. It enables shell_executor (seeding only the SHELL_TOOLING dedup key), then opts into background_agents and asserts a HARNESS ExperimentalWarning still fires — proving the shell key does not seed or suppress the HARNESS key. If shell reused the HARNESS key, the second pytest.warns would fail, so this guards the separate-dedup-key strategy against regressions.
Motivation & Context
create_harness_agenthas matured and is ready to leave experimental status, socallers can rely on it without opting into experimental APIs. However, a few of
the features it can wire in are still experimental (background agents, file
access, looping) or pre-release (shell tooling, from the alpha-stage
agent-framework-toolspackage). Graduating the factory should not silently dropthe experimental signal for those specific features. This change releases the
factory while preserving a targeted warning only for the parameters that activate
still-experimental behaviour.
Description & Review Guide
What are the major changes?
@experimental(HARNESS)decorator fromcreate_harness_agent, socalling it no longer emits an
ExperimentalWarning.ExperimentalWarningonlywhen a caller opts into a still-experimental or pre-release feature:
background_agents,file_access_store,loop_should_continue(sharedHARNESSdedup key), andshell_executor(separateSHELL_TOOLINGdedup keyso it does not suppress unrelated
HARNESSwarnings).at runtime naming the responsible parameter(s).
packages/core/AGENTS.md, andadded tests (graduation guard, default no-warning, and one warning test per
gated parameter).
What is the impact of these changes?
create_harness_agentis now a released API. This is a breaking change forpreview users who were suppressing/relying on the previous
HARNESSexperimental warning on every call.
surfaces an
ExperimentalWarning.What do you want reviewers to focus on?
HARNESSkey (so a downstream provider doesn't warn twice), while shell tooling uses a
separate
SHELL_TOOLINGkey.Related Issue
Closes #7117
This is the Python counterpart to the .NET graduation in #7119 (which references
the same issue).
Contribution Checklist
breaking changelabel (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.