test(e2e): add runtime tool-guardrail tests (Suite8cToolGuardrailsRun…#139
Open
kowser-orkes wants to merge 1 commit into
Open
test(e2e): add runtime tool-guardrail tests (Suite8cToolGuardrailsRun…#139kowser-orkes wants to merge 1 commit into
kowser-orkes wants to merge 1 commit into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. 🚀 New features to boost your workflow:
|
kowser-orkes
marked this pull request as ready for review
July 21, 2026 21:17
…time) Every prior guardrail e2e test either drives an agent-level guardrail or only inspects compiled workflow JSON for a tool-level one — nothing ever executed a tool-level guardrail. That gap is exactly how the worker- registration bug fixed by 55c74dd shipped undetected. Adds two counterfactual runtime tests against a real echo tool: - test_tool_raise_guardrail_blocks_run: an always-failing RAISE guardrail must fail/terminate the run. - test_tool_pass_guardrail_lets_run_complete: an always-passing guardrail must let the run complete, proving the worker is polled and not simply blocking unconditionally. - test_agent_fix_guardrail_substitutes_output: agent-level FIX guardrail substitutes its corrected output. Deliberately UNGATED — verified live that it already passes against today's server. It exists to catch a regression an escalate() helper to the custom-guardrail normalizer that coerces fix->raise unconditionally, without checking whether a fixed_output is present — unlike the SDK's own worker handler, which only coerces when fixedOutput == null. Applied as-is, that commit would break exactly this case. - test_tool_retry_guardrail_escalates_to_failure / test_regex_tool_guardrail_escalates_to_failure: gated behind ServerCapabilities.assumeGuardrailEscalationFix() (skip, not @disabled, until CONDUCTOR_SERVER_VERSION reaches a fixed release). Verified live with the gate forced open (GUARDRAIL_ESCALATION_FIXED=true) against today's real, unfixed server: both fail fast (~20-50s) with "Got status: COMPLETED" instead of FAILED/TERMINATED — via the SDK-worker and server-script paths respectively, confirming they'll correctly detect the fix once it ships.
kowser-orkes
force-pushed
the
feat/tool-guardrail-e2e
branch
from
July 21, 2026 21:27
209ad42 to
c30d80e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test(e2e): runtime coverage for tool guardrails + bump server pin to rc.15
Pull Request type
NOTE: Please remember to run
./gradlew spotlessApplyto fix any format violations.Changes in this PR
Describe the new behavior from this PR, and why it's needed
Suite8cToolGuardrailsRuntime(5 tests) — the first e2e coverage that actually executes atool-level guardrail. Every prior test either drove an agent-level guardrail or only inspected
compiled workflow JSON for a tool-level one, so the worker-registration bug (already fixed on
main, no client ever polled the guardrail task) shipped completely undetectedRAISEguardrail blocks the run, an always-passing one lets itcomplete — proves the tool-guardrail worker is registered and actually polled, not just present
onFail=FIXregression guard, deliberately ungated (passes today) — protectsagainst a naive server-side escalation fix discarding a valid
fixed_outputonFail=RETRYguardrail must escalate to FAILED/TERMINATEDonce
maxRetriesis exceeded instead of retrying foreverCONDUCTOR_SERVER_VERSIONinagent-e2e.yml:3.32.0-rc.8→3.32.0-rc.15, the firstrelease carrying the server-side retry-escalation fix — tests 4–5 now run for real instead of skipping
ServerCapabilitiesversion-gate helper (and its unit test) added while the fix wasunreleased — now that the pin always points at a fixed server, the gate would just be dead code
onFail=RETRY,maxRetriesexceeded3.32.0-rc.8(predates the fix)3.32.0-rc.15(first release with the fix)Issue #
Alternatives considered
Describe alternative implementation you have considered
ServerCapabilities— dropped; it was only a bridge untilthe pin moved past the fix, and keeping it after would be dead code no run ever exercises
the worker is genuinely registered and polled by the server, which a mock can't demonstrate