Skip to content

Commit af9388f

Browse files
authored
feat: ImplementAndJudge chain, per-tool error reporting, and minimum-diff judge rubric (#178)
* feat: editor bug fixes, Draft and auto save fixes. * feat(tools): add ImplementAndJudge chain tool The new chain tool wraps AgentTool twice (implementer first, then judge with the implementer's output baked into the packet) and optionally re-invokes the implementer once on NEEDS_WORK with the judge's REQUIRED FIXES section isolated under a '## Revision brief' heading. Two implementer invocations is the hard cap; higher max_revisions values clamp to one. Verdict parsing anchors on the judge's SUMMARY heading so preamble tokens ('PASS for the brief but...', 'BLOCKED would be overkill...') cannot outrank the real verdict, and missing/SUMMARY-without-token fails closed to BLOCKED rather than silently passing. The implementer's <coding_artifact> block is extracted as data (never as instructions) and passed to the judge. Required fixes are framed as untrusted feedback in the revision prompt so an embedded directive in the judge text cannot steer the write-privileged implementer. The chain refuses to launch from a non-root role, validates both child subagent types and their execution policy / required MCP servers up front, and reuses the orchestration-approval pattern from RunAgents so a session-approved chain does not re-prompt per child. * feat(toolset): surface per-tool reason in aggregated InvalidToolError Bad tool paths in agent.yaml used to surface as a bare 'Invalid tools: [...]' with the actual reason (module missing, class missing, or constructor exception) buried in the log file. The aggregated error now lists each failing tool with its per-tool reason, and a class-name miss logs a 'Did you mean <closest>?' hint. A constructor exception on one tool is now caught per-tool so the user gets one clear error naming the offending tool and the exception type instead of a bare traceback out of agent load. The whole load still aborts on any failure - agent.yaml tool references are hard requirements - but the message is now diagnosable from the traceback alone. This unblocks the stale-binary failure mode that hits users who build pythinker before a new tool lands. * feat(agents): register ImplementAndJudge, adopt minimum-diff judge rubric - default/agent.yaml: register pythinker_code.tools.agent:ImplementAndJudge alongside Agent and RunAgents so the chain is available out of the box without a custom agent spec. - default/judge.yaml: add the coding_artifact BLOCKED gate to the Context Gate (the judge must see the implementer's <coding_artifact> block when judging a code-change summary) and a Minimum-diff rubric dimension applying the reduction ladder (skip-need, reuse-stdlib, use-native, use-installed-dep, one-line, minimum) to every non-trivial diff the judge reviews. The ladder applies uniformly across review modes - no mode switch on the judge. - default/system.md: document the ImplementAndJudge default in the implementation playbook and the judge-gate guidance so parent agents prefer the chain over ad-hoc implementer+judge fan-out. - tests/core/test_agent_spec.py + test_default_agent.py: assert the new tool appears in the default agent's tool list across the spec snapshot tests so the registration does not drift. * feat(skills): bundle judge-minimum-diff and judge-overengineering-review Two static default skills replace ad-hoc prose in the system prompt with explicit, versionable content: - judge-minimum-diff: the reduction-ladder rubric the judge applies as a dimension on every non-trivial diff (skip-need, reuse-stdlib, use-native, use-installed-dep, one-line, minimum). - judge-overengineering-review: the parent-facing review checklist that walks through the same ladder before declaring a non-trivial change done. Both ship as skills/ directory entries so ReadSkill can load them on demand rather than bloating the always-on system prompt. PyInstaller datas entries are added in tests/utils/test_pyinstaller_utils.py so the PyInstaller one-file build picks them up alongside the other bundled skills. * test: cover ImplementAndJudge chain, judge branding, and toolset error reporting - tests/core/test_implement_judge_chain.py (new): unit tests for the chain's load-bearing primitives - verdict parsing (PASS / NEEDS_WORK / BLOCKED, preamble-token-resistance, missing SUMMARY fails closed to BLOCKED), artifact extraction, REQUIRED FIXES section isolation, fingerprint stability across revisions, and pydantic param validation. - tests/tools/test_implement_judge_load.py (new): regression test for the loader path - confirms ImplementAndJudgeTool instantiates through PythinkerToolset._load_tool with the same dependency-injection as AgentTool so a default-agent startup cannot regress. - tests/test_judge_branding.py (new): asserts the judge system prompt mentions the minimum-diff rubric / reduction ladder so the always-on policy does not drift away from the bundled skills. - tests/core/test_load_agent.py: extend test_load_tools_invalid to assert the aggregated error names the reason ('class or module not found'), and add test_load_tools_aggregates_constructor_errors to cover the new per-tool exception path with a monkeypatched _load_tool that raises - the same per-tool-catch handles class-miss, module-miss, and constructor-exception cases. * chore(release): add Unreleased changelog entries for chain tool, toolset error reporting, and judge rubric User-facing bullets under ## Unreleased: - InvalidToolError now names the failing tool and the reason (per-tool aggregation + 'Did you mean' hint + per-tool exception catch). - Auto-chain implementer -> judge via ImplementAndJudge tool for non-trivial scoped edits; two implementer invocations is the hard cap. - Judge adopts the minimum-diff rubric dimension applied uniformly across review modes. - Bundled judge-minimum-diff and judge-overengineering-review default skills replace ad-hoc system prompt prose. The 'changelog-entry-required' CI check requires this for any change to shipped paths; without it the PR fails before review. * fix(pr): address spell-check and brand-guard findings, drop tracked plan doc - src/pythinker_code/tools/agent/__init__.py, tests/core/test_implement_judge_chain.py: s/unparseable/unparsable/ in two docstrings. The crate-ci/typos CI check flags 'unparseable' as a typo; 'unparsable' is the standard form. - tasks/implementer-judge-chain-plan.md: delete the tracked plan doc. It leaked the upstream 'ponytail' brand and is not part of the shipped artifact. - tests/test_judge_branding.py: remove the now-deleted plan doc from _BRAND_GUARD_TARGETS so test_brand_guard_targets_exist does not flag a stale entry. * fix(agent): bound judge verdict to SUMMARY and reuse chain approval Address ImplementAndJudge review findings on PR #178: - Bound _parse_judge_verdict to the SUMMARY section so a stray PASS/NEEDS_WORK/BLOCKED token in a later section (e.g. EVIDENCE) can no longer be read as the verdict — fail closed to BLOCKED instead of leaking a false PASS on a quality gate. - Drop revision_index from the orchestration fingerprint so a NEEDS_WORK revision reuses the chain's single approval instead of re-prompting mid-chain after the implementer has already written, matching _run_agents_fingerprint. - Correct the max_revisions field docs: values above the cap are rejected at validation, not clamped. Adds regression tests for the later-section verdict leak and single-grant revision reuse. * fix(agent): fail closed on revision implementer error; refresh wire snapshot - The wire handshake snapshot (tests_e2e/test_wire_protocol.py) was stale: this PR added the judge-minimum-diff and judge-overengineering-review bundled skills but never regenerated the handshake skills list, leaving CI red on test_initialize_handshake / test_initialize_external_tool_conflict. Regenerated to include both new skills. - ImplementAndJudge: an implementer error on the revision now fails closed to BLOCKED, clearing the prior revision's stale NEEDS_WORK verdict and artifact so they can't leak into the final result (mirrors the judge-error branch). Adds a regression test. * test: make plan-mode pending-activation test hermetic test_pending_activation_returns_full asserted the plan-absent (full reminder) branch but pointed plan_path at a hardcoded /tmp/plan.md. The provider checks plan_path.exists(), so on any machine where that file happens to exist the test hit the reentry branch and failed (green only on a clean /tmp, e.g. CI). Use a tmp_path file that is never created, mirroring the sibling reentry test.
1 parent bf2f703 commit af9388f

23 files changed

Lines changed: 1770 additions & 27 deletions

File tree

CHANGELOG.md

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,64 @@ GitHub Releases page; `0.8.0` is the new starting line.
1515

1616
## Unreleased
1717

18+
- **`InvalidToolError` now names the failing tool and the reason.** A bad
19+
tool path in `agent.yaml` (typo, missing class, or — most commonly — a
20+
`pythinker` binary built before the tool was added) used to surface as a
21+
bare `Invalid tools: ['pythinker_code.tools.agent:ImplementAndJudge']`
22+
with the actual reason buried in the log file. The aggregated error now
23+
lists each bad tool with the per-tool reason, and a class-name miss logs
24+
a `Did you mean '<Closest>'?` hint. A constructor exception on one tool
25+
is caught per-tool so the user gets one clear error instead of a
26+
traceback. Rebuild the binary (`make build-bin`) if the new error names
27+
a tool that exists in the working tree.
28+
- **Auto-chain `implementer``judge` for non-trivial scoped edits.** The new
29+
`ImplementAndJudge` tool runs `implementer` once, hands the artifact to
30+
`judge`, and on `NEEDS_WORK` re-invokes `implementer` once with the
31+
judge's `REQUIRED FIXES` under a `## Revision brief` section before
32+
re-judging. Two implementer invocations is the hard cap — a
33+
still-`NEEDS_WORK` after revision surfaces the contradiction and stops.
34+
Parent agents should call `ImplementAndJudge` instead of `Agent:
35+
implementer` + `Agent: judge`; bare `judge` calls remain the right shape
36+
for non-implementation reviews (reports, audits, severity-scored
37+
findings).
38+
- **Judge adopts the minimum-diff rubric dimension.** Every non-trivial diff
39+
the judge reviews is now checked against the reduction ladder
40+
(skip-need → reuse-stdlib → use-native → use-installed-dep → one-line →
41+
minimum) and the minimum-diff rubric (no abstractions, no new deps
42+
without justification, no new config keys without a consumer, no
43+
reformatting churn outside the changed lines). The judge applies the full
44+
ladder uniformly — there is no mode switch on the judge.
45+
- **Bundled judge-branded skills.** `judge-minimum-diff` (the rubric
46+
applied as a judge dimension) and `judge-overengineering-review` (the
47+
parent-facing review checklist) ship as static default skills, replacing
48+
ad-hoc prose in the system prompt with explicit, versionable content.
49+
50+
- **Fix OpenAI Responses requests that could still send `role="system"` after
51+
switching to a newer Pythinker catalog model (gpt-5.5, gpt-5.3-codex,
52+
gpt-5.3-codex-spark, or any user-defined fine-tune).** Pythinker observed
53+
OpenAI returning `System messages are not allowed` on this path, but the
54+
`system→developer` conversion was previously gated on the openai SDK's
55+
`ResponsesModel` literal, which lags Pythinker's own model catalog. The
56+
conversion now runs unconditionally in `OpenAIResponses`, so all local
57+
system messages are normalized before sending. Also fixes the model-switch
58+
carry-over path (`_carry_context_to_session`) whose seeded `role="system"`
59+
summary message was sent verbatim on the first request after a switch.
60+
- **Background bash tasks (npm dev, docker run) no longer show the agent
61+
verb spinner.** Pure-bash background work now shows a fixed "Running in
62+
background…" label instead of "Composing…/Brewing…" verbs, which read as
63+
agent activity. Mixed bash+agent background work keeps the verb spinner
64+
while the agent is actively producing tokens.
65+
- **Quiet background tasks no longer force a 0.1s prompt repaint.** When a
66+
background task has produced no output for 2 seconds, the refresh loop
67+
drops to the idle 1.0s interval instead of spinning the braille marker at
68+
12.5 fps — fixing the "stuck spinner" look for long-running dev servers
69+
on Windows VS Code.
70+
- **Welcome banner no longer shows a stale "Update available" chip after a
71+
successful /update.** The banner chip now mirrors the under-input notice:
72+
when the update has landed this session (state=UPDATED, smoke check passed),
73+
it shows "Updated X → vY. Restart to apply." instead of telling the user
74+
to re-run an update that already completed.
75+
1876
## 0.51.0 (2026-06-22)
1977

2078
- **Reasoning levels now match each GPT model.** The thinking selector and

packages/pythinker-core/src/pythinker_core/contrib/chat_provider/openai_responses.py

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -159,10 +159,11 @@ async def generate(
159159
inputs: ResponseInputParam = []
160160
instructions = system_prompt if self._system_prompt_as_instructions else None
161161
if system_prompt and not instructions:
162-
system_message: ResponseInputItemParam = {"role": "system", "content": system_prompt}
163-
if is_openai_model(self.model_name):
164-
system_message["role"] = "developer"
165-
inputs.append(system_message)
162+
# This class is exclusively the Responses API transport (see provider.type
163+
# "openai_responses" / "openai_codex" in pythinker-code). Normalize local
164+
# system prompts to developer messages so model-name drift cannot leak an
165+
# unsupported system role onto the wire.
166+
inputs.append({"role": "developer", "content": system_prompt})
166167
# The `Message` type is OpenAI-compatible for Responses API `input` messages.
167168

168169
for message in history:
@@ -235,13 +236,13 @@ def _convert_message(self, message: Message) -> list[ResponseInputItemParam]:
235236
236237
Rules:
237238
- role in {user, assistant}: map to EasyInputMessageParam with role kept
238-
role == system: map to role=developer for OpenAI models, otherwise kept
239-
content: str kept; list[ContentPart] mapped to ResponseInputMessageContentListParam
239+
- role == system: always mapped to role=developer so model-name drift cannot
240+
leak a local system role onto the Responses API wire
240241
- role == tool: map to FunctionCallOutput with call_id and output
241242
"""
242243

243244
role = message.role
244-
if is_openai_model(self.model_name) and role == "system":
245+
if role == "system":
245246
role = "developer"
246247

247248
# tool role → function_call_output (return value from a prior tool call)

packages/pythinker-core/tests/api_snapshot_tests/test_openai_responses.py

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
import json
44
from typing import Any
55

6+
import pytest
67
import respx
78
from common import COMMON_CASES, Case, capture_request, run_test_cases
89
from httpx import Response
@@ -531,3 +532,62 @@ async def test_openai_responses_with_thinking_max_clamps_to_xhigh():
531532
pass
532533
body = json.loads(mock.calls.last.request.content.decode())
533534
assert body["reasoning"] == snapshot({"effort": "xhigh", "summary": "auto"})
535+
536+
537+
# Regression: local system-prompt roles must become developer messages regardless
538+
# of model name. A model switch to gpt-5.5 produced
539+
# `Error code: 400 - {'detail': 'System messages are not allowed'}` while the old
540+
# conversion gate was tied to the openai SDK's lagging ResponsesModel literal.
541+
@pytest.mark.parametrize(
542+
"model_name",
543+
[
544+
"gpt-5.5", # in Pythinker catalog but missing from openai SDK ResponsesModel
545+
"gpt-5.3-codex", # same — in catalog, missing from SDK
546+
"ft:gpt-5.5:my-org:custom:id", # user-defined fine-tune, never in any SDK set
547+
],
548+
)
549+
async def test_openai_responses_system_prompt_uses_developer_role(model_name: str):
550+
with respx.mock(base_url="https://api.openai.com") as mock:
551+
mock.post("/v1/responses").mock(return_value=Response(200, json=make_response()))
552+
provider = OpenAIResponses(model=model_name, api_key="test-key", stream=False)
553+
body = await capture_request(
554+
mock,
555+
provider,
556+
"You are a helpful assistant.",
557+
[],
558+
[Message(role="user", content="Hi")],
559+
)
560+
561+
assert body["input"][0] == {
562+
"role": "developer",
563+
"content": "You are a helpful assistant.",
564+
}
565+
assert all(item.get("role") != "system" for item in body["input"])
566+
567+
568+
async def test_openai_responses_history_system_message_becomes_developer():
569+
"""Mid-session model switch via `_carry_context_to_session` seeds a
570+
role='system' summary into the new session's history. With an OpenAI Responses
571+
target (e.g. gpt-5.5), that history item must be re-mapped to role='developer'
572+
on the wire — otherwise the first request after the switch is rejected.
573+
"""
574+
carried_summary = Message(
575+
role="system",
576+
content="Summary carried from the previous model session: user asked about X.",
577+
)
578+
with respx.mock(base_url="https://api.openai.com") as mock:
579+
mock.post("/v1/responses").mock(return_value=Response(200, json=make_response()))
580+
provider = OpenAIResponses(model="gpt-5.5", api_key="test-key", stream=False)
581+
body = await capture_request(
582+
mock,
583+
provider,
584+
"You are a helpful assistant.",
585+
[],
586+
[carried_summary, Message(role="user", content="continue")],
587+
)
588+
589+
roles = [item.get("role") for item in body["input"]]
590+
assert "system" not in roles, body["input"]
591+
assert roles[0] == "developer" # system_prompt
592+
assert roles[1] == "developer" # carried-over summary
593+
assert roles[2] == "user"

src/pythinker_code/agents/default/agent.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ agent:
77
tools:
88
- "pythinker_code.tools.agent:Agent"
99
- "pythinker_code.tools.agent:RunAgents"
10+
- "pythinker_code.tools.agent:ImplementAndJudge"
1011
- "pythinker_code.tools.skill:ReadSkill"
1112
# - "pythinker_code.tools.dmail:SendDMail"
1213
# - "pythinker_code.tools.think:Think"

src/pythinker_code/agents/default/judge.yaml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ agent:
1818
1919
## Context Gate
2020
- Require the parent's packet: the original request, the diff or changed files, the commands actually run with their results, residual risks, and the draft final answer. If a load-bearing piece is missing, verdict BLOCKED and name it.
21+
- When judging a code-change summary produced by `implementer`, require the implementer's `<coding_artifact>` block in the packet; if it is missing for a non-trivial code change, verdict BLOCKED and name it.
2122
2223
## External Claims (offline gate)
2324
You run offline by design — the verify profile blocks network and doc-lookup tools, so you never go online. External claims are judged by the parent's evidence, never by your own research or training-cutoff memory:
@@ -39,6 +40,7 @@ agent:
3940
- Safety and scope: no unsafe or destructive action, no secret or PII exposure, no scope creep beyond the request.
4041
- Production guardrails: changed code that touches caches, resources, trust boundaries, shared state, outbound requests, long-lived listeners, or authorization context has explicit defenses for stampedes, cleanup, schemas, races, retry storms, leaks, and IDOR risks.
4142
- Findings quality: for reports, each finding is actionable, anchored to evidence, and severity-ranked consistently with the base prompt's severity rubric (critical/high/medium/low/info).
43+
- Minimum-diff: the diff takes the smallest rung of the reduction ladder (skip-need → reuse-stdlib → use-native → use-installed-dep → one-line → minimum) before adding new code; no abstractions, dependencies, config keys, files, or error paths that the brief did not ask for. New dependencies require a one-line justification; new config keys require a one-line consumer. The ladder applies uniformly — judge once, not per-mode.
4244
4345
## Role Exit Checklist
4446
- PASS: sound; at most minor wording nits remain.

src/pythinker_code/agents/default/system.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,8 @@ Build only after requirements are understood (ask if unclear) and evidence is ga
9797

9898
For refactors, update every call site the interface change touches, and do not alter existing logic — especially in tests — beyond what the change requires. For features, add tests if the project already has tests. Migrations go additive before destructive, reversible where the framework allows; never edit a migration that already shipped. Identify the synchronization model in use and conform to it; explicitly flag any new lock, atomic, or async-boundary change. Update comments, docstrings, and README snippets your change makes false — stale documentation is a bug you just wrote.
9999

100+
**Default to `ImplementAndJudge` for non-trivial scoped edits** instead of calling `implementer` and `judge` separately. The chain runs `implementer` first, hands the artifact to `judge`, and on `NEEDS_WORK` re-invokes `implementer` once with the judge's `REQUIRED FIXES` under a `## Revision brief` section before re-judging. The implementer cap is two invocations total — a still-`NEEDS_WORK` after revision surfaces the contradiction to you and stops. Pass `scope` so the judge can `git diff` only the allowed paths; pass `acceptance` to give the judge concrete pass conditions beyond the rubric.
101+
100102
### 4.5 Research & file generation
101103

102104
For research or multimedia tasks (images, video, PDFs, docs, spreadsheets, presentations): clarify requirements first, plan before deep or wide research, design search queries deliberately. Detect tools already in the environment before installing anything; third-party installs go in an isolated/virtual environment. After generating or editing any media file, read it back to confirm the content. Never install to or delete from outside the working directory without confirmation.
@@ -121,6 +123,8 @@ For research or multimedia tasks (images, video, PDFs, docs, spreadsheets, prese
121123

122124
**Judge gate.** Before delivering high-stakes or hard-to-reverse work, run an independent `judge` subagent as the last step when available. Triggers — any one suffices: a change spanning multiple files or touching production guardrail surfaces (§6); a deliverable the user will merge, deploy, publish, or act on; a security audit or any severity-scored findings report; a release or destructive action. When unsure whether work is high-stakes, treat it as high-stakes; skip it for low-stakes, reversible, or trivial work. Hand the judge a tight packet: original request, the diff or changed files, the commands actually run with their results, residual risks, and your draft answer. It is one cheap spot-checking pass that gates your evidence — it does not redo work or replace deterministic tests and lint, so run those first. Treat `NEEDS_WORK` or `BLOCKED` as a stop: fix or revise, re-judge only if the change was material. When the judge is unavailable, walk the same checklist yourself, lead with the same `PASS`/`NEEDS_WORK`/`BLOCKED` verdict, state what verification actually ran, and put any missing packet element under **BLOCKERS**.
123125

126+
Default to `ImplementAndJudge` for non-trivial scoped edits (see §4.4); reserve a bare `judge` call for non-implementation reviews — reports, audits, severity-scored findings, or answers that don't ship code.
127+
124128
**Background shell** (root agent only). Launch long-running commands via `Shell` with `run_in_background=true` and a short `description`; the system notifies you at terminal states. `TaskList` re-enumerates active tasks (especially after context compaction); `TaskOutput` gives non-blocking snapshots (`block=true` only to intentionally wait); `TaskStop` cancels. After starting a background task, default to returning control to the user. The only task-management slash command for users is `/task` — never invent subcommands like `/task list` or `/tasks`. Subagents and sessions without these tools must not assume background-task control.
125129

126130
**Skills (`ReadSkill`).** Load a skill's exact instructions before applying its workflow — mandatory for `review-pr`, `diagnose-ci-failures`, `fix-errors`, `implement-specs`, `spec-driven-implementation`, `check-impl-against-spec`, `resolve-merge-conflicts`, and `create-pr`. Read skill details only when needed, to conserve context. Catalog and scope precedence in §12.

0 commit comments

Comments
 (0)