diff --git a/AGENTS.md b/AGENTS.md index 104af52e..467c87d5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -17,6 +17,24 @@ Pythinker CLI is a Python CLI agent for software engineering workflows. It suppo interactive shell UI, ACP server mode for IDE integrations, MCP tool loading, background work, subagents, skills, web/visualization UIs, and multi-provider LLM authentication. +## Feature Development Standard + +Build every feature as production code, not a happy-path demo. **Before implementing**, answer: +what the feature does, who or what calls it, its inputs, its outputs and side effects, how it can +fail, what happens on failure, which edge cases apply, and what test proves it works. If +requirements are ambiguous, make the safest reasonable assumption and document it — only block when +the missing detail would change the implementation. + +**Handle the failure and edge cases**, not just the happy path: missing / empty / invalid / +malformed input, unauthorized access, expired tokens, timeouts and network errors, partial success, +concurrent or duplicate requests, rate limits, large payloads, stale cache, missing records, retry +exhaustion, cancellation, and rollback/cleanup failure. Never silently ignore an unexpected state. + +**Make errors explicit**: typed or categorized, logged with actionable context, recoverable where +possible, and safe to surface — never leaking secrets, tokens, or stack traces. No bare +`except`/catch that swallows the error. For feature work this restates the Failure truthfulness +contract and C01–C15 tripwires below; ship the matching tests and verification with the feature. + ## Non-negotiable rules - **Use `uv` for Python commands.** Prefer `make ...` targets; if running tools directly, use diff --git a/CHANGELOG.md b/CHANGELOG.md index 40a223a4..15c021ee 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,46 @@ GitHub Releases page; `0.8.0` is the new starting line. ## Unreleased +- **Plugin marketplaces and activation policy.** `pythinker plugin marketplace` can add, + refresh, install, and uninstall Claude/Codex-compatible marketplace plugins. Plugins + installed for Claude Code or Codex are auto-detected (no symlink): their safe artifacts + (skills, commands, agents) activate by default, while executable artifacts (hooks, MCP + servers) stay opt-in. Config `plugins.discover_external`, `plugins.external_exec`, + `plugins.enabled`, and `plugins.disabled` — plus `pythinker plugin enable/disable ` + — control which installed plugins contribute artifacts. Hook and MCP commands + expand `${CLAUDE_PLUGIN_ROOT}`/`${PYTHINKER_PLUGIN_ROOT}` and + `${CLAUDE_PLUGIN_DATA}`/`${PYTHINKER_PLUGIN_DATA}`. +- **Plugin dependencies.** Plugins may declare `dependencies`; installing one pulls its + transitive dependencies from the same marketplace (cross-marketplace deps are blocked), + and a plugin whose dependencies aren't present+enabled is disabled at load instead of + half-activating. +- **Plugin options (`userConfig`).** `${user_config.KEY}` in a plugin's MCP server configs + and hook commands is filled from `[plugins.options.]` config; an artifact that + references an unconfigured option is skipped rather than run blank. (Content substitution, + `PYTHINKER_PLUGIN_OPTION_*` hook env vars, and keychain-backed sensitive storage are not + yet implemented.) +- **MCP tool lists refresh automatically when servers change.** Connected MCP + sessions stay open for `tools/list_changed` (and resources/prompts) notifications; + inventory is re-published without a manual `/mcp refresh`. +- **Shell live token readouts track output throughput.** The spinner and background + status line show session-wide output tokens produced during the current turn or + background stretch instead of the context-size snapshot. +- **MCP servers can be managed without a full reload.** `/mcp disconnect`, `/mcp reconnect`, and + `/mcp refresh` (or `retry`) update the live toolset for one server; disconnect unregisters its + tools and marks the server failed until reconnect. +- **Recall search matches session ids and plan slugs.** Prior-session keyword search now indexes + `session_id` and `plan_slug` in addition to titles so agents can find plan-linked sessions by slug. +- **Wire and ACP surfaces now get max-steps handoff summaries.** When a turn hits the step ceiling, + wire clients receive a streamed handoff event plus a `handoff` field on the `max_steps_reached` + result; ACP sessions emit the same summary text before returning `max_turn_requests`. +- **MCP CLI commands resolve normalized server names.** `mcp remove`, `mcp auth`, `mcp test`, and + `reset-auth` accept display names with spaces or slashes and map them to stored config keys; config + load applies the same normalization as add. +- **Compaction failure circuit breaker respects thresholds above one.** A proactive compaction + failure below `max_compaction_failures` no longer aborts the turn; the handoff fires only after + the configured number of consecutive failures. +- **AI eval gate schema is self-contained under `tests_ai/`.** Shared eval-case types live in + `tests_ai/eval_schema.py` so isolated `tests_ai` runs do not import from `tests_e2e`. - **Softer TUI chrome in the dark theme.** Panel borders (welcome banner, menus) and the input-area rules now render in a mid grey (`#8a8d91`) instead of near-white, for a less glaring look. - **Stop-time memory extraction can now be enabled explicitly.** Added an opt-in @@ -25,6 +65,28 @@ GitHub Releases page; `0.8.0` is the new starting line. `ToolSearch` plus root-session `EnterWorktree` and `ExitWorktree` tools so agents can find currently available capabilities by keyword and isolate a session's operational working directory in a git worktree without deleting user work on exit. +- **Root sessions now get a bounded git snapshot in the prompt.** When `git_status_injection` is + enabled (default), the agent receives branch, dirty-file summary, and recent commits as an + explicitly stale point-in-time reminder; disable via config or set `git_status_injection = false`. +- **Context compaction and MCP tool registration now fail more predictably.** Proactive compaction + failures hand back with an explicit `compaction_failed` stop instead of bubbling an unstructured + loop error, and MCP duplicate tool-name resolution now follows configured server order instead of + connection completion order. Tool hooks also retain the original model input even if a tool + mutates a nested argument object during execution. +- **Recall can now read bounded transcript windows.** `Recall(mode="read")` accepts + `message_offset` and `max_messages` so agents can inspect a precise, sanitized slice of a prior + workspace session without pulling the whole transcript into context. +- **MCP prompt templates can now be invoked from connected servers.** `InvokeMcpPrompt` renders a + server-published prompt with structured arguments and wraps the returned messages as untrusted + input for the model. +- **Telemetry, MCP config, and shell UX hardening.** Tool spans and metrics sanitize MCP/plugin + names; `mcp.json` load paths inject docker `--rm` and normalize server keys with collision + errors; shell suggestions accept via Alt+S into the prompt; markdown agent frontmatter maps + `max_turns`/`disallowed_tools`; `ReadMediaFile` enforces per-kind byte/pixel caps; written plans + without a Verification section get a soft warning; AI eval budgets can gate `tests_ai` reports. +- **Plan-mode exit guidance now requires verification.** The `ExitPlanMode` tool now tells agents + that written plans must include a Verification section with the smallest command, test, or check + for each meaningful change. - **Agent-loop observability now emits explicit Wire events for key runtime state.** Added `TodoListUpdated`, `SubagentToolFallback`, `AgentListDelta`, `ToolUseSkipped`, and `ContextOverflowRecovered` events, with todo updates, subagent launch fallbacks, same-step tool diff --git a/docs/en/customization/plugins.md b/docs/en/customization/plugins.md index 4d25ad10..ca13f5b1 100644 --- a/docs/en/customization/plugins.md +++ b/docs/en/customization/plugins.md @@ -314,3 +314,112 @@ Plugins and MCP servers are complementary extension mechanisms: - **MCP**: Suitable for services that need to run continuously, complex tool orchestration, or cross-process communication - **Plugins**: Suitable for simple script wrappers, project-specific tools, or rapid prototyping ::: + +## Marketplace plugins (Claude/Codex compatible) + +In addition to the script-tool plugins above, Pythinker can install and activate +**artifact plugins** from *marketplaces* — the same plugin format used by Claude +Code (`.claude-plugin/plugin.json`) and Codex. An artifact plugin contributes +skills, subagents, slash commands, hooks, and MCP servers to a session. + +### Marketplaces + +A marketplace is a catalog (`marketplace.json`) listing plugins and their +sources. Manage marketplaces with `pythinker plugin marketplace`: + +```bash +# Add a marketplace (GitHub owner/repo, git/URL, or a local path) +pythinker plugin marketplace add anthropics/claude-plugins-official +pythinker plugin marketplace add /path/to/local/marketplace --name local + +# List, refresh, or remove +pythinker plugin marketplace list +pythinker plugin marketplace refresh claude-plugins-official +pythinker plugin marketplace remove local +``` + +### Installing marketplace plugins + +```bash +# Install a plugin from a configured marketplace +pythinker plugin marketplace install ponytail@claude-plugins-official +# (equivalent positional form) +pythinker plugin marketplace install ponytail claude-plugins-official + +pythinker plugin marketplace installed # list installed marketplace plugins +pythinker plugin marketplace uninstall ponytail@claude-plugins-official +``` + +Plugins install into `~/.pythinker/plugins/cache////`. +If the same plugin or marketplace is already present in a Claude Code +(`~/.claude/plugins`) or Codex (`~/.codex/plugins`) install, Pythinker +**symlinks to it instead of copying** — no redundant downloads. + +### Activation policy + +Pythinker **auto-detects** plugins installed for Claude Code (`~/.claude/plugins`) +and Codex (`~/.codex/plugins`) — no symlink or manual copy needed. Their *safe* +artifacts (skills, commands, agents) activate automatically because they are +model-invoked, never auto-run; their *executable* artifacts (hooks, MCP servers) +auto-execute, so they stay opt-in. Detection reads the plugins in place (no +copy/symlink) and de-duplicates by name, so there is no redundancy. + +Tune this via `[plugins]` in `~/.pythinker/config.toml`: + +```toml +[plugins] +# Auto-detect Claude/Codex plugins' skills, commands, and agents. On by default. +# Set false to ignore external plugins entirely. +discover_external = true +# Also run external plugins' hooks and MCP servers (they auto-execute). Opt-in. +external_exec = false +# Empty enables all discovered plugins; a non-empty list enables only those named +# (by "name" or "name@marketplace"). +enabled = [] +# Plugins to turn off by name. Excluded even when `enabled` would allow them — +# this is how `pythinker plugin disable ` works under the all-on default. +disabled = [] +``` + +Toggle plugins without editing the file or uninstalling them: + +```bash +pythinker plugin disable ponytail # adds to [plugins].disabled +pythinker plugin enable ponytail # removes it again +``` + +Plugin-contributed hook and MCP commands may reference the plugin's own +directories via `${CLAUDE_PLUGIN_ROOT}` / `${PYTHINKER_PLUGIN_ROOT}` (the +versioned install dir) and `${CLAUDE_PLUGIN_DATA}` / `${PYTHINKER_PLUGIN_DATA}` +(a persistent per-plugin data dir); both are expanded on load. + +### Plugin dependencies + +A plugin may declare `dependencies` in its `plugin.json` (`"name"` or +`"name@marketplace"`). Installing a plugin from a marketplace also installs its +transitive dependencies from the same marketplace; cross-marketplace +dependencies are blocked (install them from their own marketplace first). At load +time, a plugin whose dependencies are not present and enabled is disabled, so it +never half-activates. + +### Plugin options (`userConfig`) + +A plugin may declare `userConfig` options and reference them as +`${user_config.KEY}` in its MCP server configs and hook commands. Provide values +per plugin in config: + +```toml +[plugins.options.my-plugin] +api_base = "https://example.test" +``` + +An MCP server or hook that references an option with no configured value is +skipped (it never runs with a blank), and the value is filled in on load. + +::: info Not yet ported +`${user_config.KEY}` substitution in **skill/agent/command content**, the +`PYTHINKER_PLUGIN_OPTION_*` **hook environment variables**, keychain-backed +storage for `sensitive` options, and an interactive enable-time prompt are not +implemented yet — they require changes outside the plugin subsystem. Today, +option values (including any marked `sensitive`) are read from config. +::: diff --git a/docs/en/release-notes/changelog.md b/docs/en/release-notes/changelog.md index 51d21c66..de5fbfaf 100644 --- a/docs/en/release-notes/changelog.md +++ b/docs/en/release-notes/changelog.md @@ -17,6 +17,48 @@ GitHub Releases page; `0.8.0` is the new starting line. ## Unreleased +- **Plugin marketplaces and activation policy.** `pythinker plugin marketplace` can add, + refresh, install, and uninstall Claude/Codex-compatible marketplace plugins. Plugins + installed for Claude Code or Codex are auto-detected (no symlink): their safe artifacts + (skills, commands, agents) activate by default, while executable artifacts (hooks, MCP + servers) stay opt-in. Config `plugins.discover_external`, `plugins.external_exec`, + `plugins.enabled`, and `plugins.disabled` — plus `pythinker plugin enable/disable ` + — control which installed plugins contribute artifacts. Hook and MCP commands + expand `${CLAUDE_PLUGIN_ROOT}`/`${PYTHINKER_PLUGIN_ROOT}` and + `${CLAUDE_PLUGIN_DATA}`/`${PYTHINKER_PLUGIN_DATA}`. +- **Plugin dependencies.** Plugins may declare `dependencies`; installing one pulls its + transitive dependencies from the same marketplace (cross-marketplace deps are blocked), + and a plugin whose dependencies aren't present+enabled is disabled at load instead of + half-activating. +- **Plugin options (`userConfig`).** `${user_config.KEY}` in a plugin's MCP server configs + and hook commands is filled from `[plugins.options.]` config; an artifact that + references an unconfigured option is skipped rather than run blank. (Content substitution, + `PYTHINKER_PLUGIN_OPTION_*` hook env vars, and keychain-backed sensitive storage are not + yet implemented.) +- **MCP tool lists refresh automatically when servers change.** Connected MCP + sessions stay open for `tools/list_changed` (and resources/prompts) notifications; + inventory is re-published without a manual `/mcp refresh`. +- **Shell live token readouts track output throughput.** The spinner and background + status line show session-wide output tokens produced during the current turn or + background stretch instead of the context-size snapshot. +- **MCP servers can be managed without a full reload.** `/mcp disconnect`, `/mcp reconnect`, and + `/mcp refresh` (or `retry`) update the live toolset for one server; disconnect unregisters its + tools and marks the server failed until reconnect. +- **Recall search matches session ids and plan slugs.** Prior-session keyword search now indexes + `session_id` and `plan_slug` in addition to titles so agents can find plan-linked sessions by slug. +- **Wire and ACP surfaces now get max-steps handoff summaries.** When a turn hits the step ceiling, + wire clients receive a streamed handoff event plus a `handoff` field on the `max_steps_reached` + result; ACP sessions emit the same summary text before returning `max_turn_requests`. +- **MCP CLI commands resolve normalized server names.** `mcp remove`, `mcp auth`, `mcp test`, and + `reset-auth` accept display names with spaces or slashes and map them to stored config keys; config + load applies the same normalization as add. +- **Compaction failure circuit breaker respects thresholds above one.** A proactive compaction + failure below `max_compaction_failures` no longer aborts the turn; the handoff fires only after + the configured number of consecutive failures. +- **AI eval gate schema is self-contained under `tests_ai/`.** Shared eval-case types live in + `tests_ai/eval_schema.py` so isolated `tests_ai` runs do not import from `tests_e2e`. +- **Softer TUI chrome in the dark theme.** Panel borders (welcome banner, menus) and the input-area + rules now render in a mid grey (`#8a8d91`) instead of near-white, for a less glaring look. - **Stop-time memory extraction can now be enabled explicitly.** Added an opt-in `memory.harvest_on_stop` setting that stages safe assistant decisions, blockers, evidence, and next steps into the existing scratchpad recall flow at turn end without writing directly to @@ -25,6 +67,28 @@ GitHub Releases page; `0.8.0` is the new starting line. `ToolSearch` plus root-session `EnterWorktree` and `ExitWorktree` tools so agents can find currently available capabilities by keyword and isolate a session's operational working directory in a git worktree without deleting user work on exit. +- **Root sessions now get a bounded git snapshot in the prompt.** When `git_status_injection` is + enabled (default), the agent receives branch, dirty-file summary, and recent commits as an + explicitly stale point-in-time reminder; disable via config or set `git_status_injection = false`. +- **Context compaction and MCP tool registration now fail more predictably.** Proactive compaction + failures hand back with an explicit `compaction_failed` stop instead of bubbling an unstructured + loop error, and MCP duplicate tool-name resolution now follows configured server order instead of + connection completion order. Tool hooks also retain the original model input even if a tool + mutates a nested argument object during execution. +- **Recall can now read bounded transcript windows.** `Recall(mode="read")` accepts + `message_offset` and `max_messages` so agents can inspect a precise, sanitized slice of a prior + workspace session without pulling the whole transcript into context. +- **MCP prompt templates can now be invoked from connected servers.** `InvokeMcpPrompt` renders a + server-published prompt with structured arguments and wraps the returned messages as untrusted + input for the model. +- **Telemetry, MCP config, and shell UX hardening.** Tool spans and metrics sanitize MCP/plugin + names; `mcp.json` load paths inject docker `--rm` and normalize server keys with collision + errors; shell suggestions accept via Alt+S into the prompt; markdown agent frontmatter maps + `max_turns`/`disallowed_tools`; `ReadMediaFile` enforces per-kind byte/pixel caps; written plans + without a Verification section get a soft warning; AI eval budgets can gate `tests_ai` reports. +- **Plan-mode exit guidance now requires verification.** The `ExitPlanMode` tool now tells agents + that written plans must include a Verification section with the smallest command, test, or check + for each meaningful change. - **Agent-loop observability now emits explicit Wire events for key runtime state.** Added `TodoListUpdated`, `SubagentToolFallback`, `AgentListDelta`, `ToolUseSkipped`, and `ContextOverflowRecovered` events, with todo updates, subagent launch fallbacks, same-step tool diff --git a/plips/plip-10-lsp-system.md b/plips/plip-10-lsp-system.md new file mode 100644 index 00000000..70ee5a7b --- /dev/null +++ b/plips/plip-10-lsp-system.md @@ -0,0 +1,753 @@ +--- +Author: Mohamed Elkholy +Updated: 2026-6-15 +Status: Proposed +--- + +# PLIP-10: Full LSP (Language Server Protocol) system for Pythinker CLI + +## Summary + +Port the reference LSP subsystem (`blackbox/pythinker-src/src/services/lsp`, +`src/tools/LSPTool`, `src/utils/plugins/lsp*`) to pythinker-code as a first-class Python +subsystem. The end state gives the agent real code intelligence — go-to-definition, +find-references, hover, document/workspace symbols, go-to-implementation, and the full call +hierarchy (prepare / incoming / outgoing) — backed by long-lived language-server processes, plus +a **passive diagnostics** stream that surfaces compiler/linter errors into the conversation after +file edits, and **plugin-based server discovery/recommendation** so users can add servers without +shipping binaries. + +The port is structured in five phases, each independently shippable and testable: + +| Phase | Deliverable | Verifies | +| --- | --- | --- | +| 0 | Dependency decision + transport (`LspClient`: JSON-RPC over stdio) | `make check-pythinker-code` + transport unit tests | +| 1 | Server instance + manager (lifecycle, routing, file sync) | manager unit tests against a fake server | +| 2 | `Lsp` agent tool — all 9 operations + formatters | tool tests + agent-spec load | +| 3 | Passive diagnostics via `DynamicInjectionProvider` | injection-provider tests | +| 4 | Plugin-based server config + recommendation | plugin-integration tests | + +This is the complete port. No operation, no lifecycle behaviour, and no diagnostic feature from +the reference is dropped. The only thing that does **not** port is the React/Ink UI layer +(recommendation menu, init-notification toasts); its *intent* is re-expressed through the existing +CLI notification + dynamic-injection systems. + +## Motivation + +* The agent currently navigates code with `Grep`/`Glob`/`SmartSearch` — textual, not semantic. It + cannot reliably answer "where is this symbol defined", "who calls this function", or "what does + the type checker say about this edit" without re-deriving it from text, which is slow and wrong + on overloads, re-exports, and dynamic dispatch. +* The reference implementation already solved this end-to-end (process management, JSON-RPC framing, + diagnostic aggregation, dedup, volume-limiting, plugin recommendation). Per the + `reference-source-of-truth` rule we port it rather than reinvent it. +* Passive diagnostics close the edit→verify loop: after the agent writes a file, the language + server's `publishDiagnostics` is surfaced into the next turn as budgeted context, so the agent + sees its own type errors without spending a tool call. +* The Host abstraction (`packages/pythinker-host`) and the dynamic-injection system + (`src/pythinker_code/soul/dynamic_injection.py`) make this a *clean* fit — the two hardest parts + (long-lived process I/O, unsolicited context injection) already have idiomatic homes. + +## The load-bearing decision: dependencies + +This is the tightest constraint and gates the whole design, so it leads. + +**Confirmed facts (verified, not assumed):** + +* There is **no** `lsprotocol`, `pygls`, `jsonrpc`, or `python-lsp-*` dependency anywhere in + `pyproject.toml` or any workspace package (`grep` over root + `packages/*` + `sdks/*` — empty). +* The ACP subsystem (`src/pythinker_code/acp/`) speaks a JSON-RPC-shaped protocol, but it **does + not hand-roll framing** — `acp/host.py:67` wraps `asyncio.StreamReader()` and delegates all + Content-Length/JSON-RPC parsing to the external `acp` library. There is therefore **no reusable + Content-Length framing code in the target** to mirror. +* New `[project].dependencies` are governed by the **zero-new-bundled-deps** policy (AGENTS.md); + adding `lsprotocol` would need explicit maintainer approval and the CONTRIBUTING justification + template. + +**Decision (recommended): hand-roll, do not add a dependency.** + +* **Framing**: LSP uses `Content-Length: N\r\n\r\n` over stdio. Reading and writing that is + ~30 lines over `HostProcess.stdin` (`AsyncWritable`) / `HostProcess.stdout` (`AsyncReadable`). + No library buys us enough to justify the supply-chain cost. +* **Types**: hand-define only the LSP types the system actually sends/receives as small Pydantic + models in `lsp/protocol.py` (≈12 models — see Phase 0). We use a *fraction* of the LSP spec; a + full `lsprotocol` type tree is dead weight. + +**Alternative (requires approval):** add `lsprotocol` (pure-Python LSP types + framing). Only +pursue if maintainers explicitly prefer spec-complete types over a hand-rolled subset. The phases +below assume the hand-rolled path; swapping in `lsprotocol` would replace `lsp/protocol.py` and the +framing half of `lsp/client.py` and leave everything else unchanged. + +## Reference architecture (what we are porting) + +Source tree (`blackbox/pythinker-src/`), ~5,400 lines of TypeScript: + +``` +src/services/lsp/ + config.ts (79) load server configs from enabled plugins + LSPClient.ts (447) JSON-RPC 2.0 client over stdio (framing, handshake, requests) + LSPServerInstance.ts (511) one server: lifecycle, health, retry, state machine + LSPServerManager.ts (420) many servers: ext→server routing, didOpen/Change/Save/Close + manager.ts (289) global singleton, lazy async init, reinit on plugin refresh + LSPDiagnosticRegistry.ts(386) store/dedup/volume-limit diagnostics, cross-turn LRU + passiveFeedback.ts (328) register publishDiagnostics handlers → registry +src/tools/LSPTool/ + LSPTool.ts (860) the agent tool: 9 ops, validation, dispatch, gitignore filter + schemas.ts (215) zod input schema (discriminated by operation) + formatters.ts (592) LSP results → human-readable text + symbolContext.ts (90) extract symbol context around a position + prompt.ts (21) tool description +src/utils/plugins/ + lspPluginIntegration.ts (387) load LSP servers from plugin manifests/.lsp.json, env resolution + lspRecommendation.ts (374) match file-ext → recommendable plugin server, install gating +src/{hooks,components}/... React UI (recommendation menu, init notifications) — intent only +``` + +### Reference behavioural contract (preserved verbatim in the port) + +These are the non-obvious behaviours that make the system robust. Each is reproduced in the port +and cited to the reference line that defines it. + +* **Spawn race guard** — after `spawn`, wait for the `spawn` event before writing to stdio, + because ENOENT (command not found) fires asynchronously; writing first yields unhandled + rejections. `LSPClient.ts:111-131`. *(Python: `Host.exec` raises synchronously on ENOENT, so the + port's equivalent is a try/except around `exec` + a post-spawn `initialize` timeout.)* +* **Initialize handshake** — `initialize` request → store `capabilities` → `initialized` + notification, with `processId`, `workspaceFolders` (LSP 3.16+, required by Pyright/gopls), + deprecated `rootUri`/`rootPath` (some servers still need them), and + `general.positionEncodings: ['utf-16']`. `LSPServerInstance.ts:167-272`. +* **Transient-error retry** — retry LSP error `-32801` ("content modified", emitted by + rust-analyzer/Pyright during indexing) up to 3× with exponential backoff (500·2^n ms). + `LSPServerInstance.ts:369-394`. +* **Crash recovery cap** — `maxRestarts` default 3; exceeding it stops retrying and parks the + server in `error`. `LSPServerInstance.ts:142-150, 314-320`. +* **Startup timeout** — wrap `initialize` in a timeout so a hung server never blocks. + `LSPServerInstance.ts:240-248, 499-511`. +* **Extension routing** — `extensionToLanguage` keys build an `ext → [serverName]` map; first + match wins. `LSPServerManager.ts:106-117, 192-207`. +* **File sync state machine** — `didOpen` tracks `fileUri → serverName`; `didChange` falls back to + `didOpen` if the file was never opened; `didSave` triggers diagnostics; `didClose` untracks. + `LSPServerManager.ts:270-400`. +* **workspace/configuration shim** — register a handler returning `[null, …]` because some servers + (tsserver) send the request even when told `configuration: false`. `LSPServerManager.ts:125-135`. +* **Lazy async singleton** — init is fire-and-forget; startup never blocks; a `generation` counter + invalidates stale init promises; reinit on plugin refresh. `manager.ts:154-253`. +* **Diagnostic dedup** — key = `{message, severity, range, source, code}`; dedup within a batch + *and* cross-turn via a 500-entry LRU keyed by file URI. `LSPDiagnosticRegistry.ts:54-56, + 110-124, 136-184`. +* **Diagnostic volume limit** — sort by severity (errors first), cap 10/file and 30 total. + `LSPDiagnosticRegistry.ts:42-43, 257-288`. +* **Diagnostic handler isolation** — the `publishDiagnostics` handler is fully wrapped in + try/except; 3 consecutive failures on a server logs a warning but never breaks the notify loop. + `passiveFeedback.ts:232-276`. +* **Tool input** — 1-based `line`/`character` (editor-style) converted to 0-based for LSP; + validate file exists + is a regular file; reject UNC paths (NTLM leak); reject files >10 MB. + `LSPTool.ts:224-414`, `schemas.ts:8-191`. +* **Result safety** — `maxResultSizeChars` 100 000; results filtered through `git check-ignore` + (batched ≤50 paths, 5 s timeout) before formatting. `LSPTool.ts`, `formatters.ts:24-72`. +* **Read-only + deferred** — `isReadOnly: true`, `shouldDefer: true` (tool disabled until LSP init + completes), `isConcurrencySafe: true`. `LSPTool.ts:127-151`. + +### LSP wire methods used (the complete set the port must speak) + +Requests (client→server): `initialize`, `textDocument/definition`, +`textDocument/references`, `textDocument/hover`, `textDocument/documentSymbol`, +`workspace/symbol`, `textDocument/implementation`, `textDocument/prepareCallHierarchy`, +`callHierarchy/incomingCalls`, `callHierarchy/outgoingCalls`, `shutdown`. +Notifications (client→server): `initialized`, `textDocument/didOpen`, `textDocument/didChange`, +`textDocument/didSave`, `textDocument/didClose`, `exit`. +Reverse (server→client): request `workspace/configuration` (shimmed); notification +`textDocument/publishDiagnostics`, `window/logMessage` (logged). + +## Target integration map (verified facts) + +Every claim below was checked against the live tree. + +* **Tool base class**: `CallableTool2[Params: BaseModel]` + (`packages/pythinker-core/src/pythinker_core/tooling/__init__.py:232`). Tools declare `name`, + `description`, `params`, `supports_parallel`; implement `async def __call__(self, params) -> + ToolReturnValue`. DI is by constructor type annotation. Example skeleton: + `src/pythinker_code/tools/web/search.py:44-75`. +* **Tool registration**: add the `module:Class` import path to the `tools:` list in + `src/pythinker_code/agents/default/agent.yaml`. Loader splits on `:`, imports, and injects + constructor deps by type from a `tool_deps` map; raise `SkipThisTool()` to disable the tool when + unavailable (`src/pythinker_code/tools/__init__.py`). Loader: + `src/pythinker_code/soul/toolset.py`; deps wired in `src/pythinker_code/soul/agent.py`. +* **Long-lived process primitive**: `Host.exec(*args, env, cwd) -> HostProcess` + (`packages/pythinker-host/src/pythinker_host/__init__.py:223-225`). `HostProcess` + (`:105-129`) exposes `stdin: AsyncWritable`, `stdout: AsyncReadable`, `stderr: AsyncReadable`, + `pid`, `returncode`, `async wait()`, `async kill()`. **This is the LSP transport substrate** — + building on it gives Local/SSH/ACP backends for free instead of raw + `asyncio.create_subprocess_exec`. +* **No reusable JSON-RPC framing** exists in target (see dependency decision). We add it. +* **Config**: `Config` is a Pydantic `BaseModel` with nested sections (`src/pythinker_code/config.py`). + Add an `lsp: LspConfig` section the same way `web`/`services` are modelled. Secret-bearing + sections are scope-locked via `SCOPE_LOCKED_PATHS`; LSP commands are not secrets, so they stay + project-overridable. +* **Approval / read-only**: a tool is read-only simply by never calling + `self._runtime.approval.request(...)`; gating happens at the execution-policy layer + (`resolve_execution_policy`, see `tools/web/search.py:62-75`). LSP queries are read-only. +* **Passive-context injection point**: `src/pythinker_code/soul/dynamic_injection.py` defines + `DynamicInjectionProvider(ABC)` with `async def get_injections(...)`, `DynamicInjection`, + `ContextBudget`/`injection_budget_from_runtime`, and `collect_within_budget`. Providers are + registered in `PythinkerSoul` (`soul/pythinkersoul.py:82-96` — `git_status`, `goal_mode`, + `agent_list`, … under `soul/dynamic_injections/`). **Passive diagnostics become one provider + here**, budget-aware by construction. `Runtime.rearm_injection` (`soul/agent.py:255-256`) lets a + tool refresh injections after an edit. +* **Plugin system**: discovery/loading under `src/pythinker_code/plugin/`; manifests are + `plugin.json`. The reference's `.lsp.json` / `manifest.lspServers` recommendation flow maps onto + this loader. +* **Tests + gate**: tool tests under `tests/tools/test_*.py` (async, fixtures). Minimum gate for + this work: `make check-pythinker-code && make test-pythinker-code`. Add a `## Unreleased` + CHANGELOG entry (required check). + +## Reference → target mapping + +| Reference (TS) | Target (Python) | Notes | +| --- | --- | --- | +| `services/lsp/LSPClient.ts` | `src/pythinker_code/lsp/client.py` | framing + handshake over `HostProcess` | +| (vscode-jsonrpc framing) | `lsp/framing.py` | hand-rolled Content-Length read/write | +| (vscode-languageserver-protocol types) | `lsp/protocol.py` | ~12 Pydantic models, only what we use | +| `services/lsp/LSPServerInstance.ts` | `lsp/instance.py` | state machine, retry, restart | +| `services/lsp/LSPServerManager.ts` | `lsp/manager.py` | routing + file sync | +| `services/lsp/manager.ts` | `lsp/service.py` | session-scoped service (not a global singleton) | +| `services/lsp/config.ts` | `lsp/config_loader.py` | merge built-in + plugin server configs | +| `services/lsp/LSPDiagnosticRegistry.ts` | `lsp/diagnostics.py` | store/dedup/volume-limit | +| `services/lsp/passiveFeedback.ts` | `lsp/diagnostics.py` (handler) + injection provider | split: capture vs surface | +| `tools/LSPTool/{LSPTool,schemas,formatters,symbolContext,prompt}.ts` | `src/pythinker_code/tools/lsp/` | the agent tool | +| `utils/plugins/lspPluginIntegration.ts` | `lsp/plugin_servers.py` | load servers from plugins | +| `utils/plugins/lspRecommendation.ts` | `lsp/recommend.py` | ext→server recommendation gating | +| React UI (hooks/components) | CLI notification + injection | intent only, no direct port | + +**Singleton → session-scoped.** The reference uses a *global* singleton (`manager.ts`) because the +TUI is one process serving one workspace. Pythinker is multi-instance and session-oriented +(`pythinker-multi-instance-invariants`), so the port owns the LSP service on the **`Runtime`** +(one service per session), constructed in `Runtime.create()` and torn down in session cleanup. No +module-global mutable state. + +## Target module layout + +``` +src/pythinker_code/lsp/ + __init__.py public: LspService, LspConfig re-exports + framing.py read_message / write_message (Content-Length over Async streams) + protocol.py Pydantic models: Position, Range, Location, Diagnostic, InitializeParams… + client.py LspClient: one process, request/notify/on_notify, handshake, shutdown + instance.py LspServerInstance: lifecycle + state machine + retry + restart + manager.py LspServerManager: ext routing + open/change/save/close file sync + service.py LspService: session-scoped facade, lazy init, status, shutdown + config_loader.py merge built-in defaults + user config + plugin servers + diagnostics.py DiagnosticRegistry (store/dedup/limit) + publishDiagnostics handler + plugin_servers.py load LSP server configs from installed plugins + recommend.py file-ext → recommendable plugin server, install gating + +src/pythinker_code/tools/lsp/ + __init__.py Lsp tool class export + tool.py Lsp(CallableTool2[Params]) — validation, dispatch, gitignore filter + schemas.py Params (discriminated by operation), 1-based→0-based + formatters.py results → text (definition/refs/hover/symbols/call-hierarchy) + symbol_context.py extract symbol context around a position + tool.md tool description (load_desc) + +src/pythinker_code/soul/dynamic_injections/ + lsp_diagnostics.py LspDiagnosticsInjectionProvider(DynamicInjectionProvider) +``` + +--- + +## Phase 0 — Dependency decision + transport + +**Goal:** a `LspClient` that can spawn a server via `Host.exec`, complete the initialize handshake, +send requests/notifications, and dispatch server notifications — with JSON-RPC framing hand-rolled. + +### Step 0.1 — `lsp/framing.py` + +Content-Length framing over the Host async streams. + +```python +# Reads/writes LSP base-protocol frames: "Content-Length: N\r\n\r\n". +async def read_message(stdout: AsyncReadable) -> dict[str, Any]: + # read header lines until blank line; parse Content-Length; read exactly N bytes; json.loads + ... + +async def write_message(stdin: AsyncWritable, message: dict[str, Any]) -> None: + body = json.dumps(message, separators=(",", ":")).encode("utf-8") + header = f"Content-Length: {len(body)}\r\n\r\n".encode("ascii") + stdin.write(header + body) + await stdin.drain() +``` + +* Use `AsyncReadable.readuntil(b"\r\n\r\n")` if available, else read line-by-line; then + `readexactly(content_length)`. Mirror whatever `acp/host.py` does with `asyncio.StreamReader` + for consistency. Reject frames with no/invalid `Content-Length` (fail-closed, typed error). +* **Tripwire C03/C06**: a malformed frame raises a typed `LspProtocolError`, never returns `{}`. + +### Step 0.2 — `lsp/protocol.py` + +Hand-define exactly the models used (do not import the LSP spec wholesale): + +`Position`, `Range`, `Location`, `LocationLink`, `Diagnostic` (+ `DiagnosticSeverity` enum), +`DocumentSymbol`, `SymbolInformation` (+ `SymbolKind` enum), `Hover`/`MarkupContent`, +`CallHierarchyItem`, `CallHierarchyIncomingCall`, `CallHierarchyOutgoingCall`, `InitializeParams`, +`InitializeResult`/`ServerCapabilities`, `PublishDiagnosticsParams`. All `pydantic.BaseModel` with +`model_config = ConfigDict(extra="ignore")` (servers send extra fields we ignore). + +### Step 0.3 — `lsp/client.py` + +```python +class LspClient: + def __init__(self, host: Host, *, logger=...): ... + @property + def capabilities(self) -> ServerCapabilities | None: ... + @property + def is_initialized(self) -> bool: ... + + async def start(self, command: str, args: list[str], *, env=None, cwd=None) -> None: + # self._proc = await host.exec(command, *args, env=env, cwd=cwd) + # spawn read loop: while returncode is None: msg = await read_message(stdout); dispatch(msg) + # ENOENT/exec failure -> LspStartError (typed). drain stderr to logger. + async def initialize(self, params: InitializeParams) -> InitializeResult: ... + async def send_request(self, method: str, params: Any) -> Any: ... # correlate by id + async def send_notification(self, method: str, params: Any) -> None: ... + def on_notification(self, method: str, handler: Callable[[Any], None]) -> None: ... + def on_request(self, method: str, handler) -> None: ... # for workspace/configuration + async def stop(self) -> None: # shutdown + exit + kill +``` + +* **Request correlation**: monotonically-increasing int `id`; a `dict[int, asyncio.Future]` pending + map; the read loop resolves the future on a matching `id`, routes `method`-only messages to + notification handlers, and answers server→client requests via `on_request` handlers. +* **Read loop**: a single `asyncio.Task` reading frames; on `returncode is not None` or + `IncompleteReadError`, fail all pending futures with `LspServerDown` (C01/C10 — never resolve a + pending request as success after the process died) and mark the client stopped. +* **`stop()`**: `send_request("shutdown")` → `send_notification("exit")` → cancel read loop → + `proc.kill()` with a short grace; idempotent; suppress errors during teardown. + +### Verification (Phase 0) + +`tests/tools/test_lsp_client.py`: a fake stdio server (an in-process `asyncio` pipe pair or a tiny +Python echo server speaking LSP framing) proves: framing round-trips; initialize handshake stores +capabilities; a request resolves on matching id; a notification reaches its handler; process death +fails pending requests with a typed error (not a hang, not a false success). Gate: +`make check-pythinker-code` + these tests. + +--- + +## Phase 1 — Server instance + manager + +### Step 1.1 — `lsp/instance.py` + +`LspServerInstance` wraps one `LspClient` with the lifecycle contract: + +```python +class LspState(StrEnum): STOPPED; STARTING; RUNNING; ERROR + +class LspServerInstance: + def __init__(self, name: str, config: LspServerConfig, host: Host): ... + state: LspState; start_time; last_error; restart_count + async def start(self) -> None: # idempotent; build InitializeParams; startup timeout; crash cap + async def stop(self) -> None: # idempotent + async def restart(self) -> None: # enforces maxRestarts + def is_healthy(self) -> bool: # RUNNING and client.is_initialized + async def send_request(self, method, params) -> Any: # retry -32801 w/ backoff (3x) + async def send_notification(self, method, params) -> None + def on_notification(self, method, handler) -> None + def on_request(self, method, handler) -> None +``` + +* `InitializeParams` built exactly as the reference (`processId=os.getpid()`, `workspaceFolders`, + `rootUri`/`rootPath`, `capabilities` with sync/hover/definition/references/documentSymbol/ + callHierarchy/publishDiagnostics, `general.positionEncodings=['utf-16']`, + `initializationOptions` from config). Cite parity to `LSPServerInstance.ts:167-237`. +* Startup timeout via `asyncio.wait_for(client.initialize(...), config.startup_timeout)`. +* Transient retry: catch LSP error code `-32801`, backoff `0.5 * 2**attempt`, ≤3 attempts. +* Crash cap: track `crash_recovery_count`; over `max_restarts` (default 3) → `ERROR`, stop retrying. + +### Step 1.2 — `lsp/manager.py` + +`LspServerManager` owns many instances and the file-sync state: + +```python +class LspServerManager: + def __init__(self, host: Host, servers: dict[str, LspServerConfig]): ... + async def initialize(self) -> None: # build ext->[server] map; register workspace/configuration shim + async def shutdown(self) -> None: # stop all (gather, isolate failures) + def server_for_file(self, path: str) -> LspServerInstance | None # ext lookup, first match + async def ensure_started(self, path: str) -> LspServerInstance | None + async def send_request(self, path, method, params) -> Any | None + async def open_file(self, path, content) -> None # didOpen, track fileUri->server + async def change_file(self, path, content) -> None # didChange, fallback to open_file + async def save_file(self, path) -> None # didSave (triggers diagnostics) + async def close_file(self, path) -> None # didClose, untrack + def is_file_open(self, path) -> bool + def all_servers(self) -> dict[str, LspServerInstance] +``` + +* `ext_map: dict[str, list[str]]` from each server's `extension_to_language` keys. +* `opened_files: dict[str, str]` (fileUri → serverName). +* `workspace/configuration` shim returns `[None] * len(params.items)`. +* Per-server init failure is isolated (continue with others); aggregate errors logged. + +### Step 1.3 — `lsp/service.py` + +Session-scoped facade (replaces the reference's global `manager.ts`): + +```python +class LspService: + @classmethod + async def create(cls, runtime: Runtime) -> "LspService": # lazy: kicks off init as a task + def status(self) -> LspInitStatus # not_started|pending|success|failed + def is_connected(self) -> bool + async def wait_for_init(self) -> None + async def reinitialize(self) -> None # on plugin refresh + async def shutdown(self) -> None + @property + def manager(self) -> LspServerManager | None + @property + def diagnostics(self) -> DiagnosticRegistry +``` + +* Init is fire-and-forget (`asyncio.create_task`); startup never blocks the agent. A `generation` + int guards against stale init completing after a reinit. +* On init success, wire the `publishDiagnostics` handlers (Phase 3). +* Held on `Runtime` (new field `lsp: LspService | None`), constructed in `Runtime.create()`, + shut down in session teardown alongside other session resources. + +### Step 1.4 — `lsp/config_loader.py` + config model + +`src/pythinker_code/config.py`: + +```python +class LspServerConfig(BaseModel): + command: str + args: list[str] = Field(default_factory=list) + extension_to_language: dict[str, str] # ".py": "python" + env: dict[str, str] = Field(default_factory=dict) + initialization_options: dict[str, Any] | None = None + startup_timeout: float = 30.0 + max_restarts: int = 3 + +class LspConfig(BaseModel): + enabled: bool = True + servers: dict[str, LspServerConfig] = Field(default_factory=dict) + +class Config(BaseModel): + ... + lsp: LspConfig = Field(default_factory=LspConfig) +``` + +* `config_loader.py` merges: built-in defaults (e.g. `pyright --stdio` for `.py` if present on + PATH) ← user `config.lsp.servers` ← plugin-provided servers (Phase 4). Later wins. +* Ship a sane built-in default for Python only (pyright if discoverable); everything else is + user/plugin-supplied. Do **not** bundle server binaries. + +TOML shape: + +```toml +[lsp] +enabled = true +[lsp.servers.python] +command = "pyright-langserver" +args = ["--stdio"] +extension_to_language = { ".py" = "python", ".pyi" = "python" } +``` + +### Verification (Phase 1) + +`tests/tools/test_lsp_manager.py` against a fake server: ext routing picks the right server; +`open_file`/`change_file` fallback chain; `didSave` emits; `shutdown` stops all and isolates a +failing server; crash cap parks at `ERROR` after `max_restarts`; transient `-32801` retries then +succeeds. Gate: `make check-pythinker-code && make test-pythinker-code`. + +--- + +## Phase 2 — The `Lsp` agent tool (all 9 operations) + +### Step 2.1 — `tools/lsp/schemas.py` + +```python +class Operation(StrEnum): + GO_TO_DEFINITION; FIND_REFERENCES; HOVER; DOCUMENT_SYMBOL; WORKSPACE_SYMBOL + GO_TO_IMPLEMENTATION; PREPARE_CALL_HIERARCHY; INCOMING_CALLS; OUTGOING_CALLS + +class Params(BaseModel): + operation: Operation + file_path: str = Field(description="File to operate on") + line: int = Field(ge=1, description="1-based line, as shown in editors") + character: int = Field(ge=1, description="1-based character offset") +``` + +Validation in the tool (not the schema, so errors are typed tool results): file exists + is a +regular file; reject UNC (`\\`/`//` prefix); reject >10 MB. Convert to 0-based LSP position +(`line-1`, `character-1`) at the boundary. + +### Step 2.2 — `tools/lsp/tool.py` + +```python +class Lsp(CallableTool2[Params]): + name = "Lsp" + description = load_desc(Path(__file__).parent / "tool.md", {}) + params = Params + supports_parallel = True # isConcurrencySafe + + def __init__(self, runtime: Runtime): + super().__init__() + if not runtime.config.lsp.enabled or runtime.lsp is None: + raise SkipThisTool() + self._runtime = runtime + self._lsp = runtime.lsp + + async def __call__(self, params: Params) -> ToolReturnValue: + builder = ToolResultBuilder() + # 0. shouldDefer: if status != success, return a clear "LSP still initializing / unavailable" + # 1. validate file (exists/regular/UNC/size) -> typed builder.error on failure + # 2. read content; manager.open_file(path, content) to ensure server has the doc + # 3. dispatch by operation -> manager.send_request(path, , position params) + # 4. None/empty -> operation-specific guidance message (not a bare empty) + # 5. gitignore-filter result paths (git check-ignore, batched<=50, 5s timeout) + # 6. format via formatters; cap at 100_000 chars + return builder.ok(text, brief=user_facing_name(params)) +``` + +Dispatch table (operation → LSP method(s)): + +| Operation | LSP call(s) | +| --- | --- | +| go_to_definition | `textDocument/definition` | +| find_references | `textDocument/references` (`context.includeDeclaration=true`) | +| hover | `textDocument/hover` | +| document_symbol | `textDocument/documentSymbol` | +| workspace_symbol | `workspace/symbol` (query="") | +| go_to_implementation | `textDocument/implementation` | +| prepare_call_hierarchy | `textDocument/prepareCallHierarchy` | +| incoming_calls | `prepareCallHierarchy` → `callHierarchy/incomingCalls` | +| outgoing_calls | `prepareCallHierarchy` → `callHierarchy/outgoingCalls` | + +* **Read-only**: never calls `approval.request`. Optionally gate on execution policy if the active + profile forbids subprocess (LSP spawns a process) — mirror `search.py`'s policy check. +* **Untrusted output**: hover/symbol text comes from project files; pass through + `ToolResultBuilder.mark_untrusted()` so it is wrapped, consistent with `utils/trust.py`. + +### Step 2.3 — `tools/lsp/formatters.py` + `symbol_context.py` + +Port `formatters.ts` faithfully: relative-path normalization (decode percent-encoding, `\`→`/`, +prefer relative if shorter and not `../../`), group references/symbols/calls by file with +`line:char`, `SymbolKind`/`DiagnosticSeverity` enum→label maps, recursive `DocumentSymbol` child +counting, call-hierarchy `fromRanges` rendering, and the exact empty-result guidance strings +(`formatters.ts:127-592`). `symbol_context.py` ports `symbolContext.ts` (extract the symbol + +surrounding lines for context). + +### Step 2.4 — `tools/lsp/tool.md` + registration + +`tool.md` = the reference `prompt.ts` text (9 operations, 1-based note, "server must be configured" +caveat). Register in `src/pythinker_code/agents/default/agent.yaml` under `tools:`: + +```yaml + - "pythinker_code.tools.lsp:Lsp" +``` + +Add the same line to any subagent spec that should have code-intelligence (e.g. `coder.yaml`, +`code_reviewer.yaml`) — decision: include in `coder`/`code-reviewer`, exclude from `explore`/`scout` +(they are textual-recon by design). Wire `Runtime` into the tool deps if not already present +(it is — `search.py` receives it). + +### Verification (Phase 2) + +`tests/tools/test_lsp_tool.py` against a fake server returning canned LSP responses: each of the 9 +operations dispatches the right method and formats correctly; 1-based→0-based conversion; UNC +rejection; >10 MB rejection; empty-result guidance; deferred behaviour when init not done; agent +spec loads with the tool present (`tests/core` agent-load test). Inline-snapshot the formatter +output (`pytest --inline-snapshot=fix`). Gate: `make check-pythinker-code && make test-pythinker-code`, +and the wire handshake snapshot in `tests_e2e/` if it pins the tool list +(`full-test-scope-includes-tests-e2e`). + +--- + +## Phase 3 — Passive diagnostics + +### Step 3.1 — `lsp/diagnostics.py` + +Port `LSPDiagnosticRegistry.ts` + the capture half of `passiveFeedback.ts`: + +```python +class DiagnosticRegistry: + def register_pending(self, server_name: str, files: list[DiagnosticFile]) -> None + def check_for_diagnostics(self) -> list[ServerDiagnostics] # dedup + volume-limit, mark sent + def clear_all(self) -> None + def clear_for_file(self, file_uri: str) -> None + def pending_count(self) -> int +``` + +* Cross-turn dedup: `OrderedDict`-based LRU (cap 500 files) keyed by file URI → set of diagnostic + keys; key = json of `{message, severity, range, source, code}`. +* Volume limit: sort by severity (errors first), cap 10/file and 30 total. +* `publishDiagnostics` handler (registered by `LspService` on init) maps LSP severity 1-4 → + Error/Warning/Info/Hint, parses URIs via `file://`→path, and `register_pending(...)`. Handler is + fully try/except-wrapped; 3 consecutive failures on a server logs once (C08 observability) but + never breaks the notify loop. + +### Step 3.2 — `soul/dynamic_injections/lsp_diagnostics.py` + +The *surface* half — a provider mirroring `git_status.py`: + +```python +class LspDiagnosticsInjectionProvider(DynamicInjectionProvider): + def __init__(self, runtime: Runtime): self._runtime = runtime + async def get_injections(self, budget: ContextBudget, ...) -> list[DynamicInjection]: + if self._runtime.lsp is None or not self._runtime.lsp.is_connected(): return [] + groups = self._runtime.lsp.diagnostics.check_for_diagnostics() + if not groups: return [] + text = render_diagnostics_block(groups) # " LSP diagnostics …" + return [DynamicInjection(text=text, ...)] # collect_within_budget truncates if needed +``` + +* Register it in `PythinkerSoul` alongside the other providers (`soul/pythinkersoul.py:82-96`). +* The provider is **budget-governed** automatically by `collect_within_budget` / + `injection_budget_from_runtime` — no separate volume logic needed beyond the registry's own caps. +* **Edit→save→diagnose loop**: after `WriteFile`/`StrReplaceFile` mutate a file, call + `runtime.lsp.manager.save_file(path)` (and `change_file` with new content) so the server + re-diagnoses, then `runtime.rearm_injection(...)` so the next turn picks up fresh diagnostics. + Wire this in the file tools' success path (a 3-line hook guarded by `runtime.lsp is not None`). + This is the single cross-cutting touch into existing tools — keep it surgical. + +### Verification (Phase 3) + +`tests/tools/test_lsp_diagnostics.py`: registry dedups within a batch and across turns; volume caps +hold; severity sort; the injection provider returns nothing when disconnected, returns a budgeted +block when diagnostics pending, and is truncated under a small budget; the file-tool hook calls +`save_file` + `rearm_injection` only when LSP is present. Gate: `make check-pythinker-code && +make test-pythinker-code`. + +--- + +## Phase 4 — Plugin-based server discovery + recommendation + +### Step 4.1 — `lsp/plugin_servers.py` + +Port `lspPluginIntegration.ts`: read LSP server configs from installed plugins via two sources — +inline `manifest.lspServers` and an external `/.lsp.json` (same schema). Resolve env +placeholders: `${PYTHINKER_PLUGIN_ROOT}`, `${PYTHINKER_PLUGIN_DATA}`, `${user_config.KEY}`, and +standard `${VAR}`. Scope names as `plugin::` to avoid collisions. Feed the result +into `config_loader.py`'s merge (highest precedence after explicit user config — decision: user +config wins over plugin, so a user can override a plugin's command). + +### Step 4.2 — `lsp/recommend.py` + +Port `lspRecommendation.ts`: on a file edit, match the extension against discoverable plugin +servers (inline manifests only — `.lsp.json` is post-install, not pre-install readable); filter by: +server supports ext ∧ binary on PATH ∧ plugin not installed ∧ not in +`config.lsp.recommendation_never` ∧ recommendations not disabled. Sort official-marketplace plugins +first. Surface **once per session**. + +### Step 4.3 — Recommendation surface (UI intent, not React) + +The reference shows a React menu (`LspRecommendationMenu.tsx`) with Yes/No/Never/Disable + +30 s auto-dismiss, and a polling init-error notification (`useLspInitializationNotification.tsx`). +Re-express the intent on the CLI: + +* **Recommendation**: emit a one-line `Suggest`-style hint (the existing `tools/suggest` / + notification path) — "Install plugin X for Python code intelligence (`pythinker plugin add X`)". + No interactive blocking menu; the agent/user acts via existing plugin commands. Track + never/disable in config (`lsp.recommendation_never: list[str]`, `lsp.recommendation_disabled: + bool`), and auto-disable after N ignores. +* **Init errors**: when `LspService.status()` is `failed` or a server is in `ERROR`, log to the + plugins-error channel surfaced by `/doctor` (dedup by `source:message`), exactly as the reference + feeds `appState.plugins.errors`. + +### Step 4.4 — Reinit on plugin refresh + +When plugins are added/removed/refreshed, call `runtime.lsp.reinitialize()` (the generation guard +makes this safe). Hook into the existing plugin-refresh path in `src/pythinker_code/plugin/`. + +### Verification (Phase 4) + +`tests/tools/test_lsp_plugins.py`: inline + `.lsp.json` server loading; env-placeholder resolution; +scope-prefixing; recommendation filter matrix (ext/binary/installed/never/disabled); official-first +sort; reinit generation guard ignores a stale init. Gate: `make check-pythinker-code && +make test-pythinker-code`. + +--- + +## Cross-cutting concerns + +* **C-tripwire compliance** (AGENTS.md): C01/C10 — process death fails pending requests, never a + false success; init failure parks the service in `failed`, never reports connected. C03/C06 — + framing/protocol errors are typed (`LspProtocolError`/`LspServerDown`), never swallowed or blurred + with empty results. C08 — the read loop and server processes have explicit lifecycle + (start/stop/restart), timeout (startup + per-request), cancellation (read-loop task cancelled on + stop), and observability (stderr drained to logger, consecutive-failure warnings). C13 — passive + diagnostics carry their `source` (server name) and are clearly framed as LSP-reported, not + agent-asserted. +* **Security**: LSP servers are subprocesses with the agent's privileges. Honour the execution + profile (don't spawn under a no-subprocess profile). Server *output* (hover/symbol text, + diagnostic messages) is untrusted project content → `mark_untrusted`. Reject UNC paths. Never log + server stdout at info (could contain source); stderr→debug only. +* **Performance**: servers are lazy (spawned on first file touch per language) and long-lived + (reused across tool calls). `documentSymbol`/`workspace/symbol` can be large → the 100 000-char + cap + spill. Diagnostics are budget-capped by the injection system. +* **Compatibility**: new config keys (`config.lsp.*`), a new tool name (`Lsp`), and a new dynamic + injection. The tool addition changes the agent's advertised tool list → update the wire-handshake + inline snapshot in `tests_e2e/` (`full-test-scope-includes-tests-e2e`). Add a `## Unreleased` + CHANGELOG bullet. No persisted-session schema change (LSP state is ephemeral per session). +* **Docs**: add an LSP page under `docs/en/customization/` (config shape, built-in Python default, + how to add a server, plugin recommendation), and update the architecture repo-map + (`docs/en/customization/architecture.md`) with the `lsp/` subsystem and its trust boundary. + +## What does NOT port + +* React/Ink components and hooks (`components/LspRecommendation`, `hooks/useLsp*`) — their *intent* + is re-expressed via the CLI notification/suggest path and `/doctor` error surface (Phase 4.3). +* The global-singleton lifetime model — replaced by session-scoped ownership on `Runtime`. + +## Open questions (resolve before/while implementing) + +1. **Built-in default servers**: ship only a Python default (pyright if on PATH), or none at all + (everything user/plugin-supplied)? Recommendation: Python-only default, gated on binary presence. +2. **`lsprotocol` vs hand-rolled**: this plan recommends hand-rolled (no new dep). Confirm with + maintainers; if they prefer `lsprotocol`, it replaces `protocol.py` + framing only. +3. **Server output trust level**: confirm `mark_untrusted` is the right wrapper for hover/symbol + text (it is project source, same trust class as `ReadFile` output). +4. **Execution-profile gating**: should LSP be disabled under read-only/review profiles (they + shouldn't spawn processes), or allowed because it's read-only? Recommendation: allow in + `coder`/default, disable where the profile forbids subprocess. + +## Verification matrix (per AGENTS.md) + +| Phase | Command | +| --- | --- | +| 0–4 (per phase) | `make check-pythinker-code && make test-pythinker-code` | +| Tool list change | rebuild wire-handshake snapshot in `tests_e2e/` (`--inline-snapshot=fix`) | +| Before PR | `## Unreleased` CHANGELOG entry; `pythinker-guard` skill; CodeRabbit green | + +## File-by-file checklist (the complete port, in build order) + +- [ ] `src/pythinker_code/lsp/framing.py` — Content-Length read/write + `LspProtocolError` +- [ ] `src/pythinker_code/lsp/protocol.py` — ~12 Pydantic models + enums +- [ ] `src/pythinker_code/lsp/client.py` — `LspClient` (spawn via `Host.exec`, handshake, requests) +- [ ] `tests/tools/test_lsp_client.py` +- [ ] `src/pythinker_code/lsp/instance.py` — `LspServerInstance` (state machine, retry, restart) +- [ ] `src/pythinker_code/lsp/manager.py` — `LspServerManager` (routing, file sync) +- [ ] `src/pythinker_code/lsp/service.py` — `LspService` (session-scoped, lazy init, reinit) +- [ ] `src/pythinker_code/lsp/config_loader.py` — merge defaults/user/plugin +- [ ] `src/pythinker_code/config.py` — add `LspConfig`/`LspServerConfig` + `Config.lsp` +- [ ] `src/pythinker_code/soul/agent.py` — `Runtime.lsp` field + construct in `Runtime.create` + teardown +- [ ] `tests/tools/test_lsp_manager.py` +- [ ] `src/pythinker_code/tools/lsp/schemas.py` +- [ ] `src/pythinker_code/tools/lsp/formatters.py` +- [ ] `src/pythinker_code/tools/lsp/symbol_context.py` +- [ ] `src/pythinker_code/tools/lsp/tool.py` — `Lsp(CallableTool2)` +- [ ] `src/pythinker_code/tools/lsp/tool.md` +- [ ] `src/pythinker_code/agents/default/agent.yaml` (+ `coder.yaml`, `code_reviewer.yaml`) +- [ ] `tests/tools/test_lsp_tool.py` +- [ ] `src/pythinker_code/lsp/diagnostics.py` — registry + publishDiagnostics handler +- [ ] `src/pythinker_code/soul/dynamic_injections/lsp_diagnostics.py` — injection provider +- [ ] `src/pythinker_code/soul/pythinkersoul.py` — register provider +- [ ] file tools (`tools/file/*`) — `save_file` + `rearm_injection` hook (guarded, surgical) +- [ ] `tests/tools/test_lsp_diagnostics.py` +- [ ] `src/pythinker_code/lsp/plugin_servers.py` +- [ ] `src/pythinker_code/lsp/recommend.py` +- [ ] `src/pythinker_code/plugin/` — reinit-on-refresh hook +- [ ] `tests/tools/test_lsp_plugins.py` +- [ ] `CHANGELOG.md` — `## Unreleased` entry +- [ ] `docs/en/customization/` — LSP page + architecture repo-map update +- [ ] `tests_e2e/` — wire-handshake snapshot refresh diff --git a/src/pythinker_code/acp/session.py b/src/pythinker_code/acp/session.py index 73eb0930..b3448a90 100644 --- a/src/pythinker_code/acp/session.py +++ b/src/pythinker_code/acp/session.py @@ -17,6 +17,7 @@ from pythinker_code.acp.types import ACPContentBlock from pythinker_code.app import PythinkerCLI from pythinker_code.soul import LLMNotSet, LLMNotSupported, MaxStepsReached, RunCancelled +from pythinker_code.soul.btw import generate_max_steps_handoff from pythinker_code.tools import extract_key_argument from pythinker_code.utils.logging import logger from pythinker_code.wire.types import ( @@ -242,6 +243,13 @@ async def prompt(self, prompt: list[ACPContentBlock]) -> acp.PromptResponse: raise acp.RequestError.internal_error({"error": str(e)}) from e except MaxStepsReached as e: logger.warning("Max steps reached: {n_steps}", n_steps=e.n_steps) + try: + handoff = await generate_max_steps_handoff(self._cli.soul) + except Exception: + logger.warning("Max-steps handoff failed", exc_info=True) + handoff = None + if handoff: + await self._send_text(f"\n── handoff ──\n{handoff}") return acp.PromptResponse(stop_reason="max_turn_requests") except RunCancelled: logger.info("Prompt cancelled by user") diff --git a/src/pythinker_code/agents/default/agent.yaml b/src/pythinker_code/agents/default/agent.yaml index ed9e089d..19aa7b9c 100644 --- a/src/pythinker_code/agents/default/agent.yaml +++ b/src/pythinker_code/agents/default/agent.yaml @@ -38,6 +38,7 @@ agent: - "pythinker_code.tools.web:FetchURL" - "pythinker_code.tools.mcp_resource:ListMcpResources" - "pythinker_code.tools.mcp_resource:ReadMcpResource" + - "pythinker_code.tools.mcp_resource:InvokeMcpPrompt" - "pythinker_code.tools.plan:ExitPlanMode" - "pythinker_code.tools.plan.enter:EnterPlanMode" subagents: diff --git a/src/pythinker_code/app.py b/src/pythinker_code/app.py index fcc0645d..78a1c285 100644 --- a/src/pythinker_code/app.py +++ b/src/pythinker_code/app.py @@ -254,6 +254,21 @@ async def create( config.loop_control.max_ralph_iterations = max_ralph_iterations logger.info("Loaded config: {config}", config=config) + # Install the plugin activation policy for this session so artifact + # collectors (skills/agents/commands/hooks/mcp) honor config-configured + # enable-state and the external (Claude/Codex) opt-in. + from pythinker_code.plugin.policy import policy_from_config, set_plugin_policy + + set_plugin_policy( + policy_from_config( + config.plugins.discover_external, + config.plugins.external_exec, + config.plugins.enabled, + config.plugins.disabled, + config.plugins.options, + ) + ) + _phase_t = time.monotonic() oauth = OAuthManager(config) @@ -392,10 +407,12 @@ async def create( # Already in plan mode from restored session, trigger activation reminder soul.schedule_plan_activation_reminder() - # Create and inject hook engine + # Create and inject hook engine. Enabled plugins contribute lifecycle + # hooks; config hooks come first so a project hook is never shadowed. from pythinker_code.hooks.engine import HookEngine + from pythinker_code.plugin.integration import plugin_hook_defs - hook_engine = HookEngine(config.hooks, cwd=str(session.work_dir)) + hook_engine = HookEngine([*config.hooks, *plugin_hook_defs()], cwd=str(session.work_dir)) if config.disabled_project_hooks: # The load-time logger.warning only reaches shell users; publish a # notification so web/ACP frontends also learn why their project diff --git a/src/pythinker_code/cli/__init__.py b/src/pythinker_code/cli/__init__.py index 38f43b65..c2625ed6 100644 --- a/src/pythinker_code/cli/__init__.py +++ b/src/pythinker_code/cli/__init__.py @@ -270,9 +270,13 @@ def _load_mcp_configs_from_cli_inputs( file_configs.append(project_mcp_file) configs: list[Any] = [] + from pythinker_code.exception import MCPConfigError + + from .mcp import prepare_mcp_config_dict + for conf in file_configs: try: - configs.append(json.loads(conf.read_text(encoding="utf-8"))) + configs.append(prepare_mcp_config_dict(json.loads(conf.read_text(encoding="utf-8")))) except json.JSONDecodeError as e: raise typer.BadParameter( f"Invalid JSON in MCP config file {conf}: {e}", @@ -283,12 +287,19 @@ def _load_mcp_configs_from_cli_inputs( f"Cannot read MCP config file {conf}: {e}", param_hint="--mcp-config-file", ) from e + except MCPConfigError as e: + raise typer.BadParameter( + f"Invalid MCP config in file {conf}: {e}", + param_hint="--mcp-config-file", + ) from e for conf in raw_mcp_config: try: - configs.append(json.loads(conf)) + configs.append(prepare_mcp_config_dict(json.loads(conf))) except json.JSONDecodeError as e: raise typer.BadParameter(f"Invalid JSON: {e}", param_hint="--mcp-config") from e + except MCPConfigError as e: + raise typer.BadParameter(f"Invalid MCP config: {e}", param_hint="--mcp-config") from e for path in _yaml_files_with_misplaced_mcp_servers(): from pythinker_code.utils.logging import logger diff --git a/src/pythinker_code/cli/mcp.py b/src/pythinker_code/cli/mcp.py index 0fa7806a..2597aa90 100644 --- a/src/pythinker_code/cli/mcp.py +++ b/src/pythinker_code/cli/mcp.py @@ -2,7 +2,7 @@ import json import os from pathlib import Path, PurePath -from typing import Annotated, Any, Literal +from typing import Annotated, Any, Literal, cast import typer @@ -27,6 +27,35 @@ def ensure_docker_rm(command: str, args: list[str]) -> list[str]: return [args[0], "--rm", *args[1:]] +def apply_docker_rm_to_mcp_config_dict(config: dict[str, Any]) -> dict[str, Any]: + """Inject ``--rm`` into stdio docker/podman servers when loading mcp.json (mcpext-3).""" + servers = config.get("mcpServers") + if not isinstance(servers, dict): + return config + typed_servers = cast(dict[str, Any], servers) + for raw_server in typed_servers.values(): + if not isinstance(raw_server, dict): + continue + server = cast(dict[str, Any], raw_server) + command = server.get("command") + if not isinstance(command, str): + continue + raw_args = server.get("args") + args: list[str] = [] + if isinstance(raw_args, list): + args = [str(item) for item in cast(list[object], raw_args)] + server["args"] = ensure_docker_rm(command, args) + return config + + +def prepare_mcp_config_dict(config: dict[str, Any]) -> dict[str, Any]: + """Apply portable MCP config fixes before validation (docker ``--rm``, name normalization).""" + from pythinker_code.utils.mcp_names import normalize_mcp_servers_in_config + + config = apply_docker_rm_to_mcp_config_dict(config) + return normalize_mcp_servers_in_config(config) + + def get_global_mcp_config_file() -> Path: """Get the global MCP config file path.""" from pythinker_code.share import get_share_dir @@ -39,6 +68,8 @@ def _load_mcp_config() -> dict[str, Any]: from fastmcp.mcp_config import MCPConfig from pydantic import ValidationError + from pythinker_code.exception import MCPConfigError + mcp_file = get_global_mcp_config_file() if not mcp_file.exists(): return {"mcpServers": {}} @@ -52,7 +83,30 @@ def _load_mcp_config() -> dict[str, Any]: except ValidationError as e: raise typer.BadParameter(f"Invalid MCP config in '{mcp_file}': {e}") from e - return config + try: + return prepare_mcp_config_dict(config) + except MCPConfigError as e: + raise typer.BadParameter(str(e)) from e + + +def _resolve_mcp_server_key(name: str, servers: dict[str, Any]) -> str: + """Resolve a user-supplied MCP server name to the stored config key.""" + from pythinker_code.exception import MCPConfigError + from pythinker_code.utils.mcp_names import normalize_mcp_server_name + + if name in servers: + return name + try: + normalized = normalize_mcp_server_name(name) + except MCPConfigError as exc: + typer.echo(str(exc), err=True) + raise typer.Exit(code=1) from exc + if normalized in servers: + if normalized != name: + typer.echo(f"Resolved MCP server '{name}' to '{normalized}'.") + return normalized + typer.echo(f"MCP server '{name}' not found.", err=True) + raise typer.Exit(code=1) def _save_mcp_config(config: dict[str, Any]) -> None: @@ -71,18 +125,23 @@ def _save_mcp_config(config: dict[str, Any]) -> None: fh.write(payload) -def _get_mcp_server(name: str, *, require_remote: bool = False) -> dict[str, Any]: - """Get MCP server config by name.""" +def _mcp_servers_from_config(config: dict[str, Any]) -> dict[str, Any]: + raw_servers = config.get("mcpServers", {}) + if isinstance(raw_servers, dict): + return cast(dict[str, Any], raw_servers) + return {} + + +def _get_mcp_server(name: str, *, require_remote: bool = False) -> tuple[str, dict[str, Any]]: + """Get MCP server config by name (accepts raw or normalized keys).""" config = _load_mcp_config() - servers = config.get("mcpServers", {}) - if name not in servers: - typer.echo(f"MCP server '{name}' not found.", err=True) - raise typer.Exit(code=1) - server = servers[name] + servers = _mcp_servers_from_config(config) + stored_name = _resolve_mcp_server_key(name, servers) + server = cast(dict[str, Any], servers[stored_name]) if require_remote and "url" not in server: - typer.echo(f"MCP server '{name}' is not a remote server.", err=True) + typer.echo(f"MCP server '{stored_name}' is not a remote server.", err=True) raise typer.Exit(code=1) - return server + return stored_name, server def _parse_key_value_pairs( @@ -171,8 +230,18 @@ def mcp_add( ] = None, ): """Add an MCP server.""" + from pythinker_code.exception import MCPConfigError + from pythinker_code.utils.mcp_names import normalize_mcp_server_name + config = _load_mcp_config() server_args = server_args or [] + try: + stored_name = normalize_mcp_server_name(name) + except MCPConfigError as exc: + typer.echo(str(exc), err=True) + raise typer.Exit(code=1) from exc + if stored_name != name: + typer.echo(f"Normalized MCP server name '{name}' to '{stored_name}'.") if transport not in {"stdio", "http"}: typer.echo(f"Unsupported transport: {transport}.", err=True) @@ -221,9 +290,12 @@ def mcp_add( if "mcpServers" not in config: config["mcpServers"] = {} - config["mcpServers"][name] = server_config + if stored_name in config["mcpServers"]: + typer.echo(f"MCP server '{stored_name}' already exists.", err=True) + raise typer.Exit(code=1) + config["mcpServers"][stored_name] = server_config _save_mcp_config(config) - typer.echo(f"Added MCP server '{name}' to {get_global_mcp_config_file()}.") + typer.echo(f"Added MCP server '{stored_name}' to {get_global_mcp_config_file()}.") @cli.command("remove") @@ -234,11 +306,12 @@ def mcp_remove( ], ): """Remove an MCP server.""" - _get_mcp_server(name) config = _load_mcp_config() - del config["mcpServers"][name] + servers = _mcp_servers_from_config(config) + stored_name = _resolve_mcp_server_key(name, servers) + del config["mcpServers"][stored_name] _save_mcp_config(config) - typer.echo(f"Removed MCP server '{name}' from {get_global_mcp_config_file()}.") + typer.echo(f"Removed MCP server '{stored_name}' from {get_global_mcp_config_file()}.") def _oauth_token_storage(server_url: str) -> Any: @@ -306,21 +379,25 @@ def mcp_auth( import asyncio server = _get_mcp_server(name, require_remote=True) - if server.get("auth") != "oauth": - typer.echo(f"MCP server '{name}' does not use OAuth. Add with --auth oauth.", err=True) + stored_name, server_config = server + if server_config.get("auth") != "oauth": + typer.echo( + f"MCP server '{stored_name}' does not use OAuth. Add with --auth oauth.", + err=True, + ) raise typer.Exit(code=1) async def _auth() -> None: import fastmcp - typer.echo(f"Authorizing with '{name}'...") + typer.echo(f"Authorizing with '{stored_name}'...") typer.echo("A browser window will open for authorization.") - client = fastmcp.Client({"mcpServers": {name: server}}) + client = fastmcp.Client({"mcpServers": {stored_name: server_config}}) try: async with client: tools = await client.list_tools() - typer.echo(f"Successfully authorized with '{name}'.") + typer.echo(f"Successfully authorized with '{stored_name}'.") typer.echo(f"Available tools: {len(tools)}") except Exception as e: typer.echo(f"Authorization failed: {type(e).__name__}: {e}", err=True) @@ -337,7 +414,7 @@ def mcp_reset_auth( ], ): """Reset OAuth authorization for an MCP server (clear cached tokens).""" - server = _get_mcp_server(name, require_remote=True) + stored_name, server = _get_mcp_server(name, require_remote=True) try: import asyncio @@ -349,7 +426,7 @@ async def _clear() -> None: await result asyncio.run(_clear()) - typer.echo(f"OAuth tokens cleared for '{name}'.") + typer.echo(f"OAuth tokens cleared for '{stored_name}'.") except ImportError: typer.echo("OAuth support not available.", err=True) raise typer.Exit(code=1) from None @@ -368,18 +445,18 @@ def mcp_test( """Test connection to an MCP server and list available tools.""" import asyncio - server = _get_mcp_server(name) + stored_name, server = _get_mcp_server(name) async def _test() -> None: import fastmcp - typer.echo(f"Testing connection to '{name}'...") - client = fastmcp.Client({"mcpServers": {name: server}}) + typer.echo(f"Testing connection to '{stored_name}'...") + client = fastmcp.Client({"mcpServers": {stored_name: server}}) try: async with client: tools = await client.list_tools() - typer.echo(f"✓ Connected to '{name}'") + typer.echo(f"✓ Connected to '{stored_name}'") typer.echo(f" Available tools: {len(tools)}") if tools: typer.echo(" Tools:") diff --git a/src/pythinker_code/cli/plugin.py b/src/pythinker_code/cli/plugin.py index 6eb7cd21..5eb4cedf 100644 --- a/src/pythinker_code/cli/plugin.py +++ b/src/pythinker_code/cli/plugin.py @@ -7,15 +7,22 @@ import socket import threading from pathlib import Path -from typing import Annotated, Any +from typing import TYPE_CHECKING, Annotated, Any from urllib.parse import urljoin, urlparse import typer from pythinker_code.plugin import PluginError +if TYPE_CHECKING: + from pythinker_code.config import Config + cli = typer.Typer(help="Manage plugins.") +# Marketplace-based plugins (Claude/Codex compatible). Kept as a subgroup so the +# legacy subprocess-tool plugin commands above stay unchanged. +marketplace_cli = typer.Typer(help="Manage plugin marketplaces and marketplace plugins.") + def _parse_git_url(target: str) -> tuple[str, str | None, str | None]: """Parse a git URL into (clone_url, subpath, branch). @@ -472,3 +479,225 @@ def info_cmd( typer.echo(f"Runtime: host={spec.runtime.host}, version={spec.runtime.host_version}") else: typer.echo("Runtime: (not installed via host)") + + +def _config_for_toggle() -> Config: + """Load config for enable/disable, requiring the default config location. + + enable/disable persist to the user ``config.toml``; refuse when the session + runs from an explicit ``--config``/``--config-file`` so we never rewrite a + file the user pointed us at ad hoc (mirrors the auth-login guard). + """ + from pythinker_code.config import load_config + + config = load_config() + if not config.is_from_default_location: + typer.echo( + "Error: enable/disable requires the default config file; " + "restart without --config/--config-file.", + err=True, + ) + raise typer.Exit(1) + return config + + +@cli.command("disable") +def disable_cmd( + name: Annotated[str, typer.Argument(help="Plugin name to disable")], +) -> None: + """Turn off a plugin (native or auto-detected) without uninstalling it.""" + from pythinker_code.config import save_config + + config = _config_for_toggle() + if name in config.plugins.disabled: + typer.echo(f"Plugin '{name}' is already disabled.") + return + config.plugins.disabled.append(name) + save_config(config) + typer.echo(f"Disabled plugin '{name}'.") + + +@cli.command("enable") +def enable_cmd( + name: Annotated[str, typer.Argument(help="Plugin name to enable")], +) -> None: + """Re-enable a previously disabled plugin.""" + from pythinker_code.config import save_config + + config = _config_for_toggle() + changed = False + if name in config.plugins.disabled: + config.plugins.disabled.remove(name) + changed = True + # If a non-empty allowlist is in force, ensure the plugin is part of it. + if config.plugins.enabled and name not in config.plugins.enabled: + config.plugins.enabled.append(name) + changed = True + if not changed: + typer.echo(f"Plugin '{name}' is already enabled.") + return + save_config(config) + typer.echo(f"Enabled plugin '{name}'.") + + +def _default_marketplace_name(source: Any) -> str: + """Derive a marketplace name from its source when none is given.""" + if source.source == "github" and source.repo: + return source.repo.rstrip("/").split("/")[-1] + if source.url: + tail = source.url.rstrip("/").split("/")[-1] + return tail[:-4] if tail.endswith(".git") else tail + if source.path: + path = Path(source.path) + return path.stem if source.source == "file" else path.name + return "marketplace" + + +@marketplace_cli.command("add") +def marketplace_add_cmd( + source: Annotated[str, typer.Argument(help="github owner/repo, git/URL, or local path")], + name: Annotated[str | None, typer.Option("--name", help="Marketplace name")] = None, +) -> None: + """Register a plugin marketplace.""" + from pythinker_code.plugin.marketplace import ( + MarketplaceError, + add_marketplace, + parse_marketplace_input, + ) + + try: + parsed = parse_marketplace_input(source) + resolved_name = name or _default_marketplace_name(parsed) + add_marketplace(resolved_name, parsed) + except MarketplaceError as exc: + typer.echo(f"Error: {exc}", err=True) + raise typer.Exit(1) from exc + typer.echo(f"Added marketplace '{resolved_name}' ({parsed.source})") + + +@marketplace_cli.command("remove") +def marketplace_remove_cmd( + name: Annotated[str, typer.Argument(help="Marketplace name")], +) -> None: + """Unregister a marketplace.""" + from pythinker_code.plugin.marketplace import remove_marketplace + + if remove_marketplace(name): + typer.echo(f"Removed marketplace '{name}'") + else: + typer.echo(f"Marketplace '{name}' not found", err=True) + raise typer.Exit(1) + + +@marketplace_cli.command("list") +def marketplace_list_cmd() -> None: + """List configured marketplaces.""" + from pythinker_code.plugin.marketplace import load_known_marketplaces + + marketplaces = load_known_marketplaces() + if not marketplaces: + typer.echo("No marketplaces configured.") + return + for name, entry in sorted(marketplaces.items()): + src = entry.source + where = src.repo or src.url or src.path or src.source + typer.echo(f" {name} ({src.source}: {where})") + + +@marketplace_cli.command("refresh") +def marketplace_refresh_cmd( + name: Annotated[str, typer.Argument(help="Marketplace name to refresh")], +) -> None: + """Re-resolve a marketplace's catalog (re-clones git sources).""" + from pythinker_code.plugin.install import refresh_marketplace + from pythinker_code.plugin.marketplace import MarketplaceError + + try: + count = refresh_marketplace(name) + except MarketplaceError as exc: + typer.echo(f"Error: {exc}", err=True) + raise typer.Exit(1) from exc + typer.echo(f"Refreshed '{name}' — {count} plugin(s) available") + + +@marketplace_cli.command("install") +def marketplace_install_cmd( + plugin: Annotated[str, typer.Argument(help="Plugin name, or name@marketplace")], + marketplace: Annotated[ + str | None, typer.Argument(help="Marketplace name (omit if using name@marketplace)") + ] = None, +) -> None: + """Install a plugin from a configured marketplace.""" + from pythinker_code.plugin.install import install_plugin_from_marketplace + from pythinker_code.plugin.marketplace import MarketplaceError + + if marketplace is None: + if "@" not in plugin: + typer.echo("Error: specify or name@marketplace", err=True) + raise typer.Exit(1) + plugin, marketplace = plugin.rsplit("@", 1) + try: + record = install_plugin_from_marketplace(plugin, marketplace) + except MarketplaceError as exc: + typer.echo(f"Error: {exc}", err=True) + raise typer.Exit(1) from exc + typer.echo(f"Installed '{plugin}@{marketplace}' v{record.version} -> {record.install_path}") + + +@marketplace_cli.command("uninstall") +def marketplace_uninstall_cmd( + plugin: Annotated[str, typer.Argument(help="Plugin name, or name@marketplace")], + marketplace: Annotated[str | None, typer.Argument(help="Marketplace name")] = None, +) -> None: + """Uninstall a marketplace plugin (removes records and cached/symlinked files).""" + import shutil + + from pythinker_code.plugin.directories import plugin_cache_dir + from pythinker_code.plugin.installed import load_installed_plugins, remove_install + + if marketplace is None: + if "@" not in plugin: + typer.echo("Error: specify or name@marketplace", err=True) + raise typer.Exit(1) + plugin, marketplace = plugin.rsplit("@", 1) + + records = load_installed_plugins().get(f"{plugin}@{marketplace}", []) + if not remove_install(plugin, marketplace): + typer.echo(f"'{plugin}@{marketplace}' is not installed", err=True) + raise typer.Exit(1) + # Remove the on-disk install (unlink symlinks; rmtree real dirs). Constrain + # deletions to the plugin cache root so corrupted metadata (an install_path + # pointing elsewhere) cannot remove arbitrary user files. Symlinks are only + # unlinked, never followed, so an external reuse target is left untouched. + cache_root = plugin_cache_dir().resolve() + for record in records: + path = Path(record.install_path) + parent = path.parent.resolve() + if parent != cache_root and cache_root not in parent.parents: + typer.echo(f"Warning: skipping unsafe uninstall path outside cache: {path}", err=True) + continue + if path.is_symlink(): + path.unlink(missing_ok=True) + elif path.is_dir(): + shutil.rmtree(path, ignore_errors=True) + plugin_dir = plugin_cache_dir() / marketplace / plugin + if plugin_dir.is_dir() and not any(plugin_dir.iterdir()): + plugin_dir.rmdir() + typer.echo(f"Uninstalled '{plugin}@{marketplace}'") + + +@marketplace_cli.command("installed") +def marketplace_installed_cmd() -> None: + """List installed marketplace plugins.""" + from pythinker_code.plugin.installed import load_installed_plugins + + plugins = load_installed_plugins() + if not plugins: + typer.echo("No marketplace plugins installed.") + return + for ident, records in sorted(plugins.items()): + versions = ", ".join(sorted({r.version for r in records})) + typer.echo(f" {ident} (v{versions})") + + +cli.add_typer(marketplace_cli, name="marketplace") diff --git a/src/pythinker_code/config.py b/src/pythinker_code/config.py index dd3d0149..aa9ddd2d 100644 --- a/src/pythinker_code/config.py +++ b/src/pythinker_code/config.py @@ -57,6 +57,9 @@ def find_project_root(cwd: Path) -> Path | None: ("providers",), # contains api_key per provider — must stay in user scope ("services",), # contains api_key fields — must stay in user scope ("feedback", "api_key"), # only the key, not the whole feedback section + # Substituted into executable plugin artifacts (MCP server configs and hook + # commands); a repo-controlled project config must not steer those values. + ("plugins", "options"), # Auto-executed when the shell starts — a repo-controlled project config # must never be able to choose the binary that runs (`command`), nor to # trigger or extend its execution (`enabled`/`segments` flip the command @@ -586,6 +589,9 @@ class LoopControl(BaseModel): """When a model response is cut off by the output-token limit and makes no tool call, nudge the model to continue at most this many times per turn before surfacing the truncated answer. ``0`` disables truncation recovery. Default: 3.""" + max_compaction_failures: int = Field(default=1, ge=1) + """Yield to the user after this many consecutive proactive compaction failures + instead of repeatedly attempting compaction. Default: 1.""" max_session_cost_usd: float | None = Field(default=None, gt=0) """Optional per-session spend ceiling in USD. When set, the turn stops with a ``budget_exhausted`` outcome once the session's accumulated estimated cost reaches @@ -1019,6 +1025,54 @@ class MCPConfig(BaseModel): ) +class PluginsConfig(BaseModel): + """Plugin/marketplace activation policy. + + Controls which installed plugins contribute artifacts (skills, agents, + commands, hooks, MCP servers) to a session. + """ + + discover_external: bool = Field( + default=True, + description=( + "Auto-detect plugins installed for Claude Code (~/.claude/plugins) and Codex " + "(~/.codex/plugins) and activate their safe artifacts (skills, commands, " + "agents). On by default — these are model-invoked, never auto-run. Set false " + "to ignore external plugins entirely." + ), + ) + external_exec: bool = Field( + default=False, + description=( + "Also run external plugins' executable artifacts (hooks and MCP servers). Off " + "by default — these auto-execute, so they are opt-in even when discover_external " + "is on." + ), + ) + enabled: list[str] = Field( + default_factory=list, + description=( + "Plugin names (or name@marketplace) to enable. Empty enables all discovered " + "plugins; a non-empty list enables only those named." + ), + ) + disabled: list[str] = Field( + default_factory=list, + description=( + "Plugin names to turn off. Excluded even when enabled would allow them — this " + "is how `pythinker plugin disable ` works under the all-on default." + ), + ) + options: dict[str, dict[str, object]] = Field( + default_factory=dict, + description=( + "Per-plugin user-config values, keyed by plugin name: " + "{plugin: {option_key: value}}. Substituted into ${user_config.KEY} " + "references in the plugin's MCP server configs and hook commands." + ), + ) + + class Config(BaseModel): """Main configuration structure.""" @@ -1090,6 +1144,13 @@ class Config(BaseModel): "Yolo mode does not inject a system reminder." ), ) + git_status_injection: bool = Field( + default=True, + description=( + "When true, inject a bounded, explicitly stale git working-tree snapshot " + "(branch, dirty summary, recent commits) into the root agent prompt at turn start." + ), + ) default_plan_mode: bool = Field(default=False, description="Default plan mode for new sessions") default_editor: str = Field( default="", @@ -1162,6 +1223,9 @@ class Config(BaseModel): description="User-submitted feedback endpoint configuration", ) mcp: MCPConfig = Field(default_factory=MCPConfig, description="MCP configuration") + plugins: PluginsConfig = Field( + default_factory=PluginsConfig, description="Plugin/marketplace activation policy" + ) tui: TUIConfig = Field(default_factory=TUIConfig, description="TUI rendering configuration") hooks: list[HookDef] = Field(default_factory=list, description="Hook definitions") # pyright: ignore[reportUnknownVariableType] disabled_project_hooks: list[str] = Field( diff --git a/src/pythinker_code/plugin/artifacts.py b/src/pythinker_code/plugin/artifacts.py new file mode 100644 index 00000000..28d394fe --- /dev/null +++ b/src/pythinker_code/plugin/artifacts.py @@ -0,0 +1,103 @@ +"""Resolve the concrete artifacts a loaded plugin contributes. + +For each artifact kind, the manifest may name explicit paths; otherwise a +convention directory under the plugin root is used (``skills/``, ``commands/``, +``agents/``, ``hooks/hooks.json``, ``.mcp.json``). All resolved paths are +constrained to the plugin root — a manifest cannot point at files outside its +own directory (path-traversal guard). +""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import cast + +from pythinker_code.plugin.loader import LoadedPlugin +from pythinker_code.utils.logging import logger + +# Convention directories/files relative to a plugin root. +_SKILLS_DIR = "skills" +_COMMANDS_DIR = "commands" +_AGENTS_DIR = "agents" +_HOOKS_FILE = "hooks/hooks.json" +_MCP_FILE = ".mcp.json" + + +def _safe_join(root: Path, relative: str) -> Path | None: + """Resolve *relative* under *root*, or None if it escapes the plugin root.""" + root_resolved = root.resolve() + candidate = (root_resolved / relative).resolve() + if candidate == root_resolved or root_resolved in candidate.parents: + return candidate + return None + + +def _resolve_dirs(plugin: LoadedPlugin, overrides: list[str], convention: str) -> list[Path]: + """Resolve a directory-valued artifact: explicit overrides or the convention dir.""" + if overrides: + resolved = [_safe_join(plugin.root, rel) for rel in overrides] + return [path for path in resolved if path is not None and path.is_dir()] + convention_dir = plugin.root / convention + return [convention_dir] if convention_dir.is_dir() else [] + + +def skill_dirs(plugin: LoadedPlugin) -> list[Path]: + """Directories containing this plugin's skills.""" + return _resolve_dirs(plugin, plugin.manifest.skills, _SKILLS_DIR) + + +def command_dirs(plugin: LoadedPlugin) -> list[Path]: + """Directories containing this plugin's slash-command prompt templates.""" + return _resolve_dirs(plugin, plugin.manifest.commands, _COMMANDS_DIR) + + +def agent_dirs(plugin: LoadedPlugin) -> list[Path]: + """Directories containing this plugin's subagent definitions.""" + return _resolve_dirs(plugin, plugin.manifest.agents, _AGENTS_DIR) + + +def hooks_file(plugin: LoadedPlugin) -> Path | None: + """Path to this plugin's hooks JSON, if it declares one. + + Inline ``hooks`` objects in the manifest are not file-backed; callers read + :attr:`PluginManifest.hooks` directly for those. This returns only a + convention/override *file* path that exists. + """ + hooks = plugin.manifest.hooks + if isinstance(hooks, str): + path = _safe_join(plugin.root, hooks) + return path if path is not None and path.is_file() else None + convention = plugin.root / _HOOKS_FILE + return convention if convention.is_file() else None + + +def mcp_servers(plugin: LoadedPlugin) -> dict[str, object]: + """MCP server configs contributed by this plugin. + + Merges the manifest ``mcpServers`` map with a convention ``.mcp.json`` file + (manifest entries win on key collision). Returns an empty dict when neither + is present or the convention file is malformed (fail-closed: a broken file + contributes nothing rather than crashing discovery). + """ + servers: dict[str, object] = {} + mcp_path = plugin.root / _MCP_FILE + if mcp_path.is_file(): + try: + raw = json.loads(mcp_path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as exc: + # Degraded behavior must be visible: a broken .mcp.json silently + # dropping the plugin's MCP servers would hide capability loss. + logger.warning( + "Ignoring unreadable plugin .mcp.json {path}: {error}", path=mcp_path, error=exc + ) + raw = None + if isinstance(raw, dict): + data = cast("dict[str, object]", raw) + # ``.mcp.json`` may nest servers under "mcpServers" or be a bare map. + inner = data.get("mcpServers", data) + if isinstance(inner, dict): + for key, value in cast("dict[str, object]", inner).items(): + servers[key] = value + servers.update(plugin.manifest.mcp_servers) + return servers diff --git a/src/pythinker_code/plugin/dependency.py b/src/pythinker_code/plugin/dependency.py new file mode 100644 index 00000000..80045702 --- /dev/null +++ b/src/pythinker_code/plugin/dependency.py @@ -0,0 +1,91 @@ +"""Plugin dependency resolution (apt-style presence checks, not an import graph). + +A dependency is a *presence guarantee*: a plugin declaring ``dependencies`` is +only activated when each dependency is also present and enabled. Mirrors the +reference design, adapted to pythinker's discovery model where a plugin is +identified by its bare ``name`` (discovery de-dups by name, so a name is unique). +Marketplace qualifiers (``name@marketplace``) are accepted in manifests but +matched by name at the discovery layer, which does not track marketplace origin. + +This module is pure: no I/O, no mutation of its inputs. +""" + +from __future__ import annotations + +from dataclasses import dataclass + + +def parse_plugin_identifier(plugin: str) -> tuple[str, str | None]: + """Split ``"name"`` or ``"name@marketplace"`` (only the first ``@`` separates).""" + if "@" in plugin: + name, _, marketplace = plugin.partition("@") + return name, marketplace or None + return plugin, None + + +@dataclass(frozen=True) +class DependencyIssue: + """One unsatisfied dependency that caused a plugin to be demoted.""" + + plugin: str + dependency: str + reason: str # "not-enabled" (known but off) | "not-found" (absent everywhere) + + def message(self) -> str: + if self.reason == "not-enabled": + return ( + f'Plugin "{self.plugin}" disabled: dependency "{self.dependency}" ' + "is installed but not enabled." + ) + return ( + f'Plugin "{self.plugin}" disabled: dependency "{self.dependency}" ' + "was not found in any configured plugin." + ) + + +def verify_and_demote( + names_with_deps: list[tuple[str, list[str]]], enabled: set[str] +) -> tuple[set[str], list[DependencyIssue]]: + """Disable plugins whose declared dependencies are not satisfied. + + Fixed-point: demoting a plugin can break a dependent that required it, so the + scan repeats until no further demotions occur (monotone — a plugin only ever + leaves the enabled set — so it terminates). + + Args: + names_with_deps: ``(plugin_name, [dependency_ref, ...])`` for every known + plugin (enabled or not). Dependency refs are matched by name. + enabled: names currently enabled (subject to demotion). + + Returns the set of demoted names and one :class:`DependencyIssue` per + unsatisfied dependency. ``reason`` distinguishes a dependency that is known + (installed) but disabled from one that is absent entirely. + """ + known = {name for name, _ in names_with_deps} + active = set(enabled) + demoted: set[str] = set() + issues: list[DependencyIssue] = [] + + changed = True + while changed: + changed = False + for name, deps in names_with_deps: + if name not in active: + continue + for dep in deps: + dep_name = parse_plugin_identifier(dep)[0] + if dep_name in active: + continue + active.discard(name) + demoted.add(name) + issues.append( + DependencyIssue( + plugin=name, + dependency=dep, + reason="not-enabled" if dep_name in known else "not-found", + ) + ) + changed = True + break # stop scanning this plugin's deps; restart the outer pass + + return demoted, issues diff --git a/src/pythinker_code/plugin/directories.py b/src/pythinker_code/plugin/directories.py new file mode 100644 index 00000000..430cd437 --- /dev/null +++ b/src/pythinker_code/plugin/directories.py @@ -0,0 +1,94 @@ +"""On-disk locations for the plugin/marketplace system. + +pythinker owns ``~/.pythinker/plugins/``; for cross-ecosystem compatibility the +loader also reads (read-only) the Claude Code and Codex plugin roots, so plugins +already installed for those tools (e.g. a Claude ``ponytail`` plugin) contribute +their artifacts without re-installation. +""" + +from __future__ import annotations + +from pathlib import Path + +from pythinker_code.plugin.manager import get_plugins_dir + +# --- pythinker-owned state ------------------------------------------------- + + +def known_marketplaces_file() -> Path: + """Registry of configured marketplaces.""" + return get_plugins_dir() / "known_marketplaces.json" + + +def installed_plugins_file() -> Path: + """Installation metadata for plugins installed from marketplaces.""" + return get_plugins_dir() / "installed_plugins.json" + + +def marketplaces_cache_dir() -> Path: + """Cached marketplace manifests (``.json`` or cloned ``/``).""" + return get_plugins_dir() / "marketplaces" + + +def plugin_cache_dir() -> Path: + """Installed plugin contents, keyed ``///``.""" + return get_plugins_dir() / "cache" + + +def plugin_data_dir(plugin_name: str) -> Path: + """Per-plugin persistent data directory (survives updates). Created lazily.""" + return get_plugins_dir() / "data" / plugin_name + + +# --- cross-ecosystem read-only roots --------------------------------------- + + +def claude_plugin_roots() -> list[Path]: + """Claude Code plugin install roots (cache/local), if present.""" + base = Path.home() / ".claude" / "plugins" + return [base / "cache", base / "local"] + + +def codex_plugin_roots() -> list[Path]: + """Codex plugin install root, if present.""" + return [Path.home() / ".codex" / "plugins"] + + +def external_plugin_roots() -> list[Path]: + """All read-only third-party plugin roots scanned for compatibility.""" + return claude_plugin_roots() + codex_plugin_roots() + + +def _external_cache_bases() -> list[Path]: + """Versioned plugin-cache bases for Claude and Codex (``//``).""" + return [ + Path.home() / ".claude" / "plugins" / "cache", + Path.home() / ".codex" / "plugins" / "cache", + ] + + +def external_installed_plugin_dirs(marketplace: str, plugin: str) -> list[Path]: + """Existing Claude/Codex install dirs for ``marketplace/plugin`` (any version). + + Used to avoid redundant copies: if another tool already has the plugin on + disk, pythinker symlinks to it instead of re-fetching. + """ + found: list[Path] = [] + for base in _external_cache_bases(): + plugin_dir = base / marketplace / plugin + if not plugin_dir.is_dir(): + continue + try: + found.extend(sorted(v for v in plugin_dir.iterdir() if v.is_dir())) + except OSError: + continue + return found + + +def external_marketplace_dirs(name: str) -> list[Path]: + """Existing Claude/Codex marketplace clones named *name*.""" + candidates = [ + Path.home() / ".claude" / "plugins" / "marketplaces" / name, + Path.home() / ".codex" / "plugins" / "marketplaces" / name, + ] + return [c for c in candidates if c.is_dir()] diff --git a/src/pythinker_code/plugin/install.py b/src/pythinker_code/plugin/install.py new file mode 100644 index 00000000..bc649c55 --- /dev/null +++ b/src/pythinker_code/plugin/install.py @@ -0,0 +1,417 @@ +"""Install a plugin from a configured marketplace into the versioned cache. + +Resolves the marketplace manifest (local file/directory, or fetched git/url), +finds the named plugin entry, materializes its source into +``cache////``, and records the install. Plugin and +marketplace names are sanitized before they touch the filesystem, and copied +sources are confined to the marketplace root (no ``../`` escape). + +ponytail: git uses a shallow ``git clone`` subprocess (works offline against a +local repo path); npm/pip plugin sources are not yet supported and raise a clear +error rather than silently doing nothing. +""" + +from __future__ import annotations + +import re +import shutil +import subprocess +import tempfile +from datetime import UTC +from pathlib import Path +from typing import cast + +from pythinker_code.plugin.dependency import parse_plugin_identifier +from pythinker_code.plugin.directories import ( + external_installed_plugin_dirs, + external_marketplace_dirs, + marketplaces_cache_dir, + plugin_cache_dir, +) +from pythinker_code.plugin.installed import ( + InstalledRecord, + load_installed_plugins, + plugin_identifier, + record_install, + remove_install, +) +from pythinker_code.plugin.manifest import ( + MarketplaceEntry, + MarketplaceManifest, + PluginManifestError, + find_marketplace_manifest, + find_plugin_manifest, + load_marketplace_manifest, + load_plugin_manifest, +) +from pythinker_code.plugin.marketplace import ( + KnownMarketplaceEntry, + MarketplaceError, + MarketplaceSource, + load_known_marketplaces, + resolve_local_marketplace, + save_known_marketplaces, +) +from pythinker_code.utils.logging import logger + +_SAFE_NAME = re.compile(r"^[A-Za-z0-9._-]+$") +_GIT_TIMEOUT_S = 120 +# Allowed git transports. Excludes exec-capable transports (``ext::``, ``fd::``) +# so a marketplace-controlled URL cannot smuggle a command, and excludes the +# plaintext ``http://``/``git://`` transports (unauthenticated, tamperable in +# transit — a supply-chain risk for executable plugin content). ``file://`` is +# permitted for local/offline clones. +_SAFE_GIT_URL = re.compile(r"^(https://|ssh://|git@|file://)") + + +def _safe_name(name: str, kind: str) -> str: + """Reject names that are unsafe as a path segment.""" + if not _SAFE_NAME.match(name) or name in {".", ".."}: + raise MarketplaceError(f"Unsafe {kind} name: {name!r}") + return name + + +def _git_clone(url: str, ref: str | None, dest: Path) -> None: + """Shallow-clone *url* into *dest*. Raises :class:`MarketplaceError` on failure. + + Hardened against argv flag-smuggling and exec-capable transports: the ref may + not begin with ``-``, the URL must use an allowed scheme, and ``--`` ends + option parsing before the positional ``url``/``dest`` reach git. + """ + if not _SAFE_GIT_URL.match(url): + raise MarketplaceError(f"Unsafe or unsupported git URL: {url!r}") + cmd = ["git", "clone", "--depth", "1"] + if ref: + if ref.startswith("-"): + raise MarketplaceError(f"Unsafe git ref: {ref!r}") + cmd += ["--branch", ref] + cmd += ["--", url, str(dest)] + try: + result = subprocess.run( + cmd, + capture_output=True, + text=True, + encoding="utf-8", + errors="replace", + timeout=_GIT_TIMEOUT_S, + check=False, + ) + except (OSError, subprocess.SubprocessError) as exc: + raise MarketplaceError(f"git clone failed for {url}: {exc}") from exc + if result.returncode != 0: + raise MarketplaceError(f"git clone failed for {url}: {result.stderr.strip()}") + + +def _remove_path(path: Path) -> None: + """Remove a file, symlink, or directory, surfacing failures. + + ``shutil.rmtree`` refuses symlinks, so a leftover symlink at a cache dest + would survive an ``ignore_errors`` cleanup and then break the next clone/copy. + Handle symlinks explicitly and raise (don't silently ignore) on failure. + """ + try: + if path.is_symlink() or path.is_file(): + path.unlink(missing_ok=True) + elif path.is_dir(): + shutil.rmtree(path) + except OSError as exc: + raise MarketplaceError(f"Could not remove existing path {path}: {exc}") from exc + + +def _symlink(dest: Path, target: Path) -> None: + """Point *dest* at *target* via a directory symlink (replacing any existing dest).""" + dest.parent.mkdir(parents=True, exist_ok=True) + if dest.is_symlink() or dest.exists(): + if dest.is_dir() and not dest.is_symlink(): + shutil.rmtree(dest, ignore_errors=True) + else: + dest.unlink(missing_ok=True) + dest.symlink_to(target, target_is_directory=True) + + +def _reuse_external_plugin( + marketplace_name: str, plugin_name: str, dest_parent: Path +) -> Path | None: + """If Claude/Codex already has this plugin, symlink to it instead of fetching. + + Returns the symlinked dest dir (named for the reused version) or None when no + external install exists. Avoids duplicating plugin contents on disk. + """ + for external in external_installed_plugin_dirs(marketplace_name, plugin_name): + if find_plugin_manifest(external) is None: + continue + dest = dest_parent / external.name # reuse the external version label + _symlink(dest, external) + logger.info("Reusing external plugin install via symlink: {target}", target=external) + return dest + return None + + +def _marketplace_repo_url(source: MarketplaceSource) -> str: + if source.source == "github": + if not source.repo: + raise MarketplaceError("github marketplace source missing repo") + return f"https://github.com/{source.repo}.git" + if source.source == "git": + if not source.url: + raise MarketplaceError("git marketplace source missing url") + return source.url + raise MarketplaceError(f"Cannot fetch marketplace source '{source.source}'") + + +def _load_marketplace(name: str, entry: KnownMarketplaceEntry) -> tuple[MarketplaceManifest, Path]: + """Return (manifest, marketplace_root) for a configured marketplace.""" + source = entry.source + if source.source in ("file", "directory"): + manifest = resolve_local_marketplace(source) + if source.source == "file": + root = Path(source.path).parent if source.path else Path() + else: + root = Path(source.path or "") + return manifest, root + if source.source in ("github", "git"): + dest = marketplaces_cache_dir() / _safe_name(name, "marketplace") + # No redundancy: reuse a Claude/Codex clone of this marketplace if present. + external = external_marketplace_dirs(name) + if external: + _symlink(dest, external[0]) + else: + _remove_path(dest) + _git_clone(_marketplace_repo_url(source), source.ref, dest) + manifest_path = find_marketplace_manifest(dest) + if manifest_path is None: + raise MarketplaceError(f"No marketplace.json in fetched marketplace '{name}'") + return load_marketplace_manifest(manifest_path), dest + raise MarketplaceError(f"Unsupported marketplace source '{source.source}'") + + +def _find_entry(manifest: MarketplaceManifest, plugin_name: str) -> MarketplaceEntry: + for entry in manifest.plugins: + if entry.name == plugin_name: + return entry + raise MarketplaceError(f"Plugin '{plugin_name}' not found in marketplace '{manifest.name}'") + + +def _materialize_plugin_source( + entry: MarketplaceEntry, marketplace_root: Path, dest: Path +) -> str | None: + """Copy/clone the plugin's source into *dest*. Returns a git SHA when known.""" + source = entry.source + if isinstance(source, str): + # Relative path under the marketplace repo root. + resolved = (marketplace_root / source).resolve() + root_resolved = marketplace_root.resolve() + if resolved != root_resolved and root_resolved not in resolved.parents: + raise MarketplaceError(f"Plugin source escapes marketplace root: {source}") + if not resolved.is_dir(): + raise MarketplaceError(f"Plugin source not found: {resolved}") + shutil.copytree(resolved, dest, ignore=shutil.ignore_patterns(".git")) + return None + if isinstance(source, dict): + source_d = cast("dict[str, object]", source) + kind = source_d.get("source") + if kind in ("github", "git"): + url = ( + f"https://github.com/{source_d.get('repo')}.git" + if kind == "github" + else source_d.get("url") + ) + if not url: + raise MarketplaceError("git plugin source missing url/repo") + if not isinstance(url, str): + raise MarketplaceError("git plugin source url must be a string") + ref = source_d.get("ref") + tmp = Path(tempfile.mkdtemp(prefix="pythinker-plugin-")) + try: + _git_clone(url, ref if isinstance(ref, str) else None, tmp / "repo") + shutil.copytree(tmp / "repo", dest, ignore=shutil.ignore_patterns(".git")) + finally: + shutil.rmtree(tmp, ignore_errors=True) + return None + raise MarketplaceError(f"Unsupported plugin source kind: {kind!r}") + raise MarketplaceError(f"Plugin '{entry.name}' has no usable source") + + +def resolve_marketplace_catalog(name: str) -> MarketplaceManifest: + """Load a configured marketplace's catalog (fetching git/url sources).""" + marketplaces = load_known_marketplaces() + if name not in marketplaces: + raise MarketplaceError(f"Marketplace not configured: {name}") + manifest, _root = _load_marketplace(name, marketplaces[name]) + return manifest + + +def refresh_marketplace(name: str) -> int: + """Re-resolve a marketplace and stamp its ``lastUpdated``. Returns plugin count.""" + from datetime import datetime + + marketplaces = load_known_marketplaces() + if name not in marketplaces: + raise MarketplaceError(f"Marketplace not configured: {name}") + manifest, _root = _load_marketplace(name, marketplaces[name]) + marketplaces[name].last_updated = datetime.now(UTC).isoformat() + save_known_marketplaces(marketplaces) + return len(manifest.plugins) + + +def install_plugin_from_marketplace( + plugin_name: str, marketplace_name: str, *, scope: str = "user" +) -> InstalledRecord: + """Install ``plugin_name`` from ``marketplace_name`` into the versioned cache. + + Raises :class:`MarketplaceError` if the marketplace is not configured, the + plugin is absent, or its source cannot be materialized. + """ + plugin_name = _safe_name(plugin_name, "plugin") + marketplace_name = _safe_name(marketplace_name, "marketplace") + + known = load_known_marketplaces() + if marketplace_name not in known: + raise MarketplaceError(f"Marketplace not configured: {marketplace_name}") + + # No redundancy: if Claude/Codex already has this plugin on disk, symlink to + # it and skip resolving/fetching the marketplace entirely. + reused = _reuse_external_plugin( + marketplace_name, plugin_name, plugin_cache_dir() / marketplace_name / plugin_name + ) + if reused is not None: + record = InstalledRecord(scope=scope, installPath=str(reused), version=reused.name) + record_install(plugin_name, marketplace_name, record) + return record + + manifest, marketplace_root = _load_marketplace(marketplace_name, known[marketplace_name]) + installed: dict[str, InstalledRecord] = {} + # Records materialized so far, oldest first. If dependency resolution fails + # partway through (e.g. a cross-marketplace dep), every plugin already written + # to disk/registry is rolled back so a failed install never leaves partial state. + materialized: list[tuple[str, InstalledRecord]] = [] + try: + _install_with_deps( + plugin_name, + marketplace_name, + manifest, + marketplace_root, + scope, + installed, + [], + materialized, + ) + except Exception: + _rollback_installs(marketplace_name, materialized) + raise + return installed[plugin_name] + + +def _rollback_installs( + marketplace_name: str, materialized: list[tuple[str, InstalledRecord]] +) -> None: + """Undo partially completed installs (best-effort), newest first. + + Runs while unwinding a failed install, so a cleanup error must not mask the + original failure — each step is logged and continued rather than raised. + """ + for name, record in reversed(materialized): + ident = plugin_identifier(name, marketplace_name) + try: + _remove_path(Path(record.install_path)) + remove_install(name, marketplace_name, scope=record.scope) + except Exception as exc: + logger.warning( + "Could not fully roll back partial install of {id}: {error}", + id=ident, + error=exc, + ) + + +def _materialize_and_record( + plugin_name: str, + marketplace_name: str, + manifest: MarketplaceManifest, + marketplace_root: Path, + scope: str, +) -> InstalledRecord: + """Fetch one plugin's source into the versioned cache and record the install.""" + entry = _find_entry(manifest, plugin_name) + version = _safe_name(entry.version or "unknown", "version") + + dest = plugin_cache_dir() / marketplace_name / plugin_name / version + _remove_path(dest) + dest.parent.mkdir(parents=True, exist_ok=True) + + try: + sha = _materialize_plugin_source(entry, marketplace_root, dest) + except Exception: + shutil.rmtree(dest, ignore_errors=True) # don't leave a half-written plugin + raise + + record = InstalledRecord(scope=scope, installPath=str(dest), version=version, gitCommitSha=sha) + record_install(plugin_name, marketplace_name, record) + logger.info( + "Installed plugin {id} -> {dest}", + id=plugin_identifier(plugin_name, marketplace_name), + dest=dest, + ) + return record + + +def _installed_manifest_deps(install_path: Path) -> list[str]: + """Read a just-installed plugin's normalized ``dependencies`` (fail-soft).""" + try: + return load_plugin_manifest(install_path).dependencies + except PluginManifestError: + return [] + + +def _install_with_deps( + plugin_name: str, + marketplace_name: str, + manifest: MarketplaceManifest, + marketplace_root: Path, + scope: str, + installed: dict[str, InstalledRecord], + in_progress: list[str], + materialized: list[tuple[str, InstalledRecord]], +) -> None: + """Install a plugin and its transitive dependencies from the same marketplace. + + Dependencies are resolved from each plugin's own ``plugin.json`` after it is + materialized (marketplace entries don't carry them). Cross-marketplace + dependencies are blocked (install them from their own marketplace first), and + cycles are detected via the active install path. Already-installed + dependencies are skipped. + """ + if plugin_name in installed: + return + if plugin_name in in_progress: + cycle = " -> ".join([*in_progress, plugin_name]) + raise MarketplaceError(f"Plugin dependency cycle: {cycle}") + + in_progress.append(plugin_name) + record = _materialize_and_record( + plugin_name, marketplace_name, manifest, marketplace_root, scope + ) + materialized.append((plugin_name, record)) + for dep in _installed_manifest_deps(Path(record.install_path)): + dep_name, dep_marketplace = parse_plugin_identifier(dep) + if dep_marketplace is not None and dep_marketplace != marketplace_name: + raise MarketplaceError( + f"Cross-marketplace dependency '{dep}' of '{plugin_name}' is not allowed; " + f"install it from '{dep_marketplace}' first." + ) + dep_name = _safe_name(dep_name, "plugin") + if dep_name in installed or plugin_identifier(dep_name, marketplace_name) in ( + load_installed_plugins() + ): + continue + _install_with_deps( + dep_name, + marketplace_name, + manifest, + marketplace_root, + scope, + installed, + in_progress, + materialized, + ) + in_progress.remove(plugin_name) + installed[plugin_name] = record diff --git a/src/pythinker_code/plugin/installed.py b/src/pythinker_code/plugin/installed.py new file mode 100644 index 00000000..fc635efd --- /dev/null +++ b/src/pythinker_code/plugin/installed.py @@ -0,0 +1,128 @@ +"""Installed-plugin registry (``installed_plugins.json``, schema v2). + +Tracks which plugins are installed from marketplaces, keyed by the +``name@marketplace`` identifier, each with one or more scoped install records. +Mirrors the reference's v2 layout. I/O is fail-soft and atomic. +""" + +from __future__ import annotations + +import json +import os +from pathlib import Path +from typing import Any, cast + +from pydantic import BaseModel, ConfigDict, Field + +from pythinker_code.plugin.directories import installed_plugins_file +from pythinker_code.utils.io import file_lock +from pythinker_code.utils.logging import logger + +INSTALLED_SCHEMA_VERSION = 2 + + +class InstalledRecord(BaseModel): + """One scoped installation of a plugin.""" + + model_config = ConfigDict(extra="ignore", populate_by_name=True) + + scope: str = "user" + install_path: str = Field(alias="installPath") + version: str = "unknown" + installed_at: str | None = Field(default=None, alias="installedAt") + last_updated: str | None = Field(default=None, alias="lastUpdated") + git_commit_sha: str | None = Field(default=None, alias="gitCommitSha") + + +def plugin_identifier(name: str, marketplace: str) -> str: + """Canonical ``name@marketplace`` identifier.""" + return f"{name}@{marketplace}" + + +def _atomic_write_json(path: Path, data: Any) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_suffix(path.suffix + ".tmp") + tmp.write_text(json.dumps(data, ensure_ascii=False, indent=2), encoding="utf-8") + os.replace(tmp, path) + + +def load_installed_plugins() -> dict[str, list[InstalledRecord]]: + """Load the install registry, skipping malformed records (fail-soft).""" + path = installed_plugins_file() + if not path.is_file(): + return {} + try: + raw = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as exc: + logger.warning("Cannot read installed_plugins.json: {error}", error=exc) + return {} + if not isinstance(raw, dict): + return {} + plugins_raw = cast("dict[str, Any]", raw).get("plugins", {}) + if not isinstance(plugins_raw, dict): + return {} + result: dict[str, list[InstalledRecord]] = {} + for ident, records in cast("dict[str, Any]", plugins_raw).items(): + if not isinstance(records, list): + continue + parsed: list[InstalledRecord] = [] + for record in cast("list[Any]", records): + try: + parsed.append(InstalledRecord.model_validate(record)) + except Exception as exc: + logger.warning("Skipping invalid install record for {id}: {e}", id=ident, e=exc) + if parsed: + result[ident] = parsed + return result + + +def save_installed_plugins(plugins: dict[str, list[InstalledRecord]]) -> None: + """Persist the install registry atomically in v2 layout.""" + payload = { + "version": INSTALLED_SCHEMA_VERSION, + "plugins": { + ident: [r.model_dump(by_alias=True, exclude_none=True) for r in records] + for ident, records in plugins.items() + }, + } + _atomic_write_json(installed_plugins_file(), payload) + + +def record_install(name: str, marketplace: str, record: InstalledRecord) -> None: + """Add or replace an install record for ``name@marketplace`` in its scope. + + The load → mutate → save runs under a cross-process lock so concurrent + ``pythinker plugin`` invocations (or a session overlapping a CLI install) + can't drop each other's registry updates. + """ + with file_lock(installed_plugins_file()): + plugins = load_installed_plugins() + ident = plugin_identifier(name, marketplace) + existing = [r for r in plugins.get(ident, []) if r.scope != record.scope] + plugins[ident] = [*existing, record] + save_installed_plugins(plugins) + + +def remove_install(name: str, marketplace: str, *, scope: str | None = None) -> bool: + """Remove install records for a plugin (optionally only one scope). + + Returns True if anything was removed. The read-modify-write runs under the + same cross-process lock as :func:`record_install`. + """ + with file_lock(installed_plugins_file()): + plugins = load_installed_plugins() + ident = plugin_identifier(name, marketplace) + if ident not in plugins: + return False + if scope is None: + del plugins[ident] + else: + kept = [r for r in plugins[ident] if r.scope != scope] + if len(kept) == len(plugins[ident]): + return False + if kept: + plugins[ident] = kept + else: + del plugins[ident] + save_installed_plugins(plugins) + return True diff --git a/src/pythinker_code/plugin/integration.py b/src/pythinker_code/plugin/integration.py new file mode 100644 index 00000000..505ee7a1 --- /dev/null +++ b/src/pythinker_code/plugin/integration.py @@ -0,0 +1,251 @@ +"""Bridge installed plugins into pythinker's artifact-discovery paths. + +Each collector runs one discovery pass and maps the enabled plugins to the +concrete artifact locations the rest of the app already knows how to consume. +This keeps the wiring in the host modules (``skill``, ``subagents``, ``soul``) +to a single call each. Collectors are added here as each consumer is wired. + +Trust posture: external (Claude/Codex) plugins contribute executable agents, +hooks, and MCP servers, so they must not auto-activate just by being installed +for another tool. Collectors therefore default to pythinker-owned plugins only; +opting into external plugins (and per-plugin enable-state) is config-gated and +wired in a later phase. + +ponytail: discovery is a bounded filesystem walk over a few roots, so each +collector re-runs it rather than sharing a cached pass. Add a session-scoped +cache only if startup profiling shows it matters. +""" + +from __future__ import annotations + +import json +from collections.abc import Callable +from functools import partial +from pathlib import Path +from typing import Any, cast + +from pydantic import ValidationError + +from pythinker_code.hooks.config import HOOK_EVENT_TYPES, HookDef +from pythinker_code.plugin import artifacts +from pythinker_code.plugin.directories import plugin_data_dir +from pythinker_code.plugin.loader import LoadedPlugin, discover_plugins +from pythinker_code.plugin.options import UserConfigError, substitute_user_config_vars +from pythinker_code.plugin.policy import PluginPolicy, current_plugin_policy +from pythinker_code.utils.logging import logger + + +def _enabled_plugins(policy: PluginPolicy | None, *, include_external: bool) -> list[LoadedPlugin]: + """Discover the plugins this session activates under *policy* (or the active one).""" + pol = policy if policy is not None else current_plugin_policy() + enabled_set = set(pol.enabled) if pol.enabled is not None else None + plugins = discover_plugins(include_external=include_external, is_enabled=enabled_set).enabled + if pol.disabled: + plugins = [plugin for plugin in plugins if plugin.name not in pol.disabled] + return plugins + + +def _safe_external(policy: PluginPolicy | None) -> bool: + """Whether safe (model-invoked) artifacts may come from external plugins.""" + pol = policy if policy is not None else current_plugin_policy() + return pol.discover_external + + +def _exec_external(policy: PluginPolicy | None) -> bool: + """Whether executable (auto-running) artifacts may come from external plugins.""" + pol = policy if policy is not None else current_plugin_policy() + return pol.discover_external and pol.external_exec + + +def _safe_artifact_dirs( + policy: PluginPolicy | None, extract: Callable[[LoadedPlugin], list[Path]] +) -> list[Path]: + """Collect a safe (model-invoked) artifact's dirs across enabled plugins.""" + dirs: list[Path] = [] + for plugin in _enabled_plugins(policy, include_external=_safe_external(policy)): + dirs.extend(extract(plugin)) + return dirs + + +def plugin_skill_dirs(policy: PluginPolicy | None = None) -> list[Path]: + """Skill roots contributed by enabled plugins (each a ``skills/``-style dir).""" + return _safe_artifact_dirs(policy, artifacts.skill_dirs) + + +def plugin_agent_dirs(policy: PluginPolicy | None = None) -> list[Path]: + """Subagent-definition roots contributed by enabled plugins (``agents/`` dirs).""" + return _safe_artifact_dirs(policy, artifacts.agent_dirs) + + +def plugin_command_dirs(policy: PluginPolicy | None = None) -> list[Path]: + """Slash-command prompt-template roots contributed by enabled plugins.""" + return _safe_artifact_dirs(policy, artifacts.command_dirs) + + +def _expand_plugin_vars(text: str, plugin: LoadedPlugin) -> str: + """Expand ``${...PLUGIN_ROOT}`` / ``${...PLUGIN_DATA}`` to this plugin's dirs. + + Both the Claude (``CLAUDE_*``) and pythinker (``PYTHINKER_*``) spellings are + accepted for cross-ecosystem compatibility. + + Trust boundary: the paths are interpolated unquoted, mirroring the reference + (which runs hooks via a shell too). Safe because native plugin roots are + sanitized at install (``_SAFE_NAME``) so they cannot hold shell metacharacters, + and external (Claude/Codex) hooks/MCP only run when ``external_exec`` is opted + in. Do not add shlex.quote here: it would diverge and break ``${ROOT}/x --flag``. + """ + root = str(plugin.root) + data = str(plugin_data_dir(plugin.name)) + return ( + text.replace("${CLAUDE_PLUGIN_ROOT}", root) + .replace("${PYTHINKER_PLUGIN_ROOT}", root) + .replace("${CLAUDE_PLUGIN_DATA}", data) + .replace("${PYTHINKER_PLUGIN_DATA}", data) + ) + + +def _map_strings(value: object, transform: Callable[[str], str]) -> object: + """Apply *transform* to every string in a nested config value.""" + if isinstance(value, str): + return transform(value) + if isinstance(value, list): + return [_map_strings(item, transform) for item in cast("list[object]", value)] + if isinstance(value, dict): + return { + key: _map_strings(val, transform) + for key, val in cast("dict[str, object]", value).items() + } + return value + + +def _plugin_options(plugin: LoadedPlugin, policy: PluginPolicy | None) -> dict[str, object]: + """User-config values configured for *plugin* (empty if none).""" + pol = policy if policy is not None else current_plugin_policy() + return pol.options.get(plugin.name, {}) + + +def plugin_mcp_servers(policy: PluginPolicy | None = None) -> dict[str, object]: + """MCP server configs contributed by enabled plugins (earlier plugins win). + + Executable artifact: external plugins contribute only when ``external_exec``. + ``${...PLUGIN_ROOT}``/``${...PLUGIN_DATA}`` are expanded so a plugin can point + at its own bundled server, and ``${user_config.KEY}`` is filled from config. A + server referencing an unconfigured option is skipped (fail-soft), not run blank. + """ + servers: dict[str, object] = {} + for plugin in _enabled_plugins(policy, include_external=_exec_external(policy)): + # Always validate ${user_config.*} references, even when the manifest + # declares no userConfig: an unresolved placeholder must fail-soft (skip + # the artifact) rather than reach an executable command literally. + options = _plugin_options(plugin, policy) + for key, value in artifacts.mcp_servers(plugin).items(): + expanded = _map_strings(value, partial(_expand_plugin_vars, plugin=plugin)) + try: + expanded = _map_strings( + expanded, partial(substitute_user_config_vars, values=options) + ) + except UserConfigError as exc: + logger.warning( + "Skipping MCP server {key} from {plugin}: unconfigured user_config {error}", + key=key, + plugin=plugin.name, + error=exc, + ) + continue + servers.setdefault(key, expanded) + return servers + + +def _hooks_payload(plugin: LoadedPlugin) -> dict[str, Any] | None: + """Return a plugin's raw hooks mapping (inline manifest object or hooks.json).""" + inline = plugin.manifest.hooks + if isinstance(inline, dict): + return cast("dict[str, Any]", inline) + hooks_path = artifacts.hooks_file(plugin) + if hooks_path is None: + return None + try: + data = json.loads(hooks_path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as exc: + logger.warning( + "Skipping unreadable plugin hooks {path}: {error}", path=hooks_path, error=exc + ) + return None + return cast("dict[str, Any]", data) if isinstance(data, dict) else None + + +def _translate_hook_defs( + plugin: LoadedPlugin, payload: dict[str, Any], options: dict[str, object] +) -> list[HookDef]: + """Translate a Claude-style hooks mapping into pythinker ``HookDef`` entries. + + Shape: ``{event: [{matcher, hooks: [{type: command, command, timeout}]}]}``, + optionally wrapped in a top-level ``"hooks"`` key. Only ``command`` hooks are + supported; ``${...PLUGIN_ROOT}``/``${...PLUGIN_DATA}`` expand to the plugin's + dirs and ``${user_config.KEY}`` is filled from *options* (a hook referencing an + unconfigured option is skipped). Malformed entries are skipped, never fatal. + """ + raw = payload.get("hooks", payload) + if not isinstance(raw, dict): + return [] + defs: list[HookDef] = [] + for event, groups in cast("dict[str, Any]", raw).items(): + if event not in HOOK_EVENT_TYPES or not isinstance(groups, list): + continue + for group in cast("list[Any]", groups): + if not isinstance(group, dict): + continue + group_d = cast("dict[str, Any]", group) + matcher = group_d.get("matcher", "") + entries = group_d.get("hooks", []) + if not isinstance(entries, list): + continue + for entry in cast("list[Any]", entries): + if not isinstance(entry, dict): + continue + entry_d = cast("dict[str, Any]", entry) + command = entry_d.get("command") + if entry_d.get("type", "command") != "command" or not isinstance(command, str): + continue + expanded = _expand_plugin_vars(command, plugin) + try: + expanded = substitute_user_config_vars(expanded, options) + except UserConfigError as exc: + logger.warning( + "Skipping {event} hook in {plugin}: unconfigured user_config {error}", + event=event, + plugin=plugin.name, + error=exc, + ) + continue + timeout = entry_d.get("timeout") + try: + defs.append( + HookDef( + event=cast("Any", event), + command=expanded, + matcher=matcher if isinstance(matcher, str) else "", + **({"timeout": timeout} if isinstance(timeout, int) else {}), + ) + ) + except ValidationError as exc: + logger.warning( + "Skipping invalid plugin hook in {plugin}: {error}", + plugin=plugin.name, + error=exc, + ) + return defs + + +def plugin_hook_defs(policy: PluginPolicy | None = None) -> list[HookDef]: + """Lifecycle hooks contributed by enabled plugins, as pythinker ``HookDef``s. + + Executable artifact: external plugins contribute only when ``external_exec``. + """ + defs: list[HookDef] = [] + for plugin in _enabled_plugins(policy, include_external=_exec_external(policy)): + payload = _hooks_payload(plugin) + if payload is not None: + options = _plugin_options(plugin, policy) + defs.extend(_translate_hook_defs(plugin, payload, options)) + return defs diff --git a/src/pythinker_code/plugin/loader.py b/src/pythinker_code/plugin/loader.py new file mode 100644 index 00000000..c207c967 --- /dev/null +++ b/src/pythinker_code/plugin/loader.py @@ -0,0 +1,178 @@ +"""Discover and load artifact-contributing plugins across ecosystems. + +A plugin root is any directory whose ``plugin.json`` is resolvable via +:func:`pythinker_code.plugin.manifest.find_plugin_manifest`. Discovery walks the +pythinker plugin cache plus the read-only Claude/Codex roots, identifies plugin +roots at any reasonable depth (marketplaces nest as +``///``), and loads each manifest. + +Discovery is fail-open at the *collection* level — one malformed plugin never +aborts the scan — but fail-closed per plugin: a plugin that fails to parse is +recorded as an error and contributes nothing. +""" + +from __future__ import annotations + +from collections.abc import Iterator +from dataclasses import dataclass, replace +from pathlib import Path +from typing import Literal + +from pythinker_code.plugin.directories import ( + claude_plugin_roots, + codex_plugin_roots, + plugin_cache_dir, +) +from pythinker_code.plugin.manifest import ( + PluginManifest, + PluginManifestError, + find_plugin_manifest, + load_plugin_manifest, +) + +PluginOrigin = Literal["pythinker", "claude", "codex", "builtin"] + +# Plugins nest at most a few levels under a root (marketplace/plugin/version). +_MAX_DISCOVERY_DEPTH = 4 + + +@dataclass(frozen=True) +class LoadedPlugin: + """A successfully loaded plugin and where it came from.""" + + name: str + root: Path + manifest: PluginManifest + origin: PluginOrigin + enabled: bool = True + + +@dataclass(frozen=True) +class PluginLoadError: + """A plugin root that could not be loaded.""" + + root: Path + message: str + + +@dataclass(frozen=True) +class PluginLoadResult: + """Outcome of a discovery pass: loaded plugins plus per-plugin errors.""" + + plugins: list[LoadedPlugin] + errors: list[PluginLoadError] + + @property + def enabled(self) -> list[LoadedPlugin]: + return [p for p in self.plugins if p.enabled] + + +def _iter_plugin_roots(base: Path, *, max_depth: int = _MAX_DISCOVERY_DEPTH) -> Iterator[Path]: + """Yield directories under *base* that contain a plugin manifest. + + Stops descending once a plugin root is found (no nested plugins inside a + plugin) and skips dot-directories as descent targets (the manifest's own + ``.claude-plugin``/``.pythinker-plugin`` dir is still inspected on the parent). + """ + if not base.is_dir(): + return + stack: list[tuple[Path, int]] = [(base, 0)] + while stack: + current, depth = stack.pop() + if find_plugin_manifest(current) is not None: + yield current + continue + if depth >= max_depth: + continue + try: + # Sort for deterministic discovery: filesystem iteration order is + # unspecified, and first-wins dedupe downstream must not depend on it + # (C11 — no non-determinism in execution-critical paths). + children = sorted( + c for c in current.iterdir() if c.is_dir() and not c.name.startswith(".") + ) + except OSError: + continue + for child in children: + stack.append((child, depth + 1)) + + +def _roots_by_origin() -> list[tuple[Path, PluginOrigin]]: + roots: list[tuple[Path, PluginOrigin]] = [(plugin_cache_dir(), "pythinker")] + for root in claude_plugin_roots(): + roots.append((root, "claude")) + for root in codex_plugin_roots(): + roots.append((root, "codex")) + return roots + + +def discover_plugins( + *, + include_external: bool = True, + is_enabled: set[str] | None = None, +) -> PluginLoadResult: + """Discover and load all plugins. + + Args: + include_external: also scan the read-only Claude/Codex roots. + is_enabled: if given, only plugins whose name is in the set are marked + enabled (others load but are disabled). ``None`` enables all — the + pre-marketplace default until per-scope enable-state lands. + + Later origins never override an already-loaded plugin name: the pythinker + cache wins over Claude/Codex, so a plugin installed natively takes priority. + """ + plugins: list[LoadedPlugin] = [] + errors: list[PluginLoadError] = [] + seen: set[str] = set() + roots: list[tuple[Path, PluginOrigin]] = _roots_by_origin() + if not include_external: + roots = [pair for pair in roots if pair[1] == "pythinker"] + + for base, origin in roots: + for plugin_root in _iter_plugin_roots(base): + try: + manifest = load_plugin_manifest(plugin_root) + except PluginManifestError as exc: + errors.append(PluginLoadError(root=plugin_root, message=str(exc))) + continue + if manifest.name in seen: + continue + seen.add(manifest.name) + enabled = True if is_enabled is None else manifest.name in is_enabled + plugins.append( + LoadedPlugin( + name=manifest.name, + root=plugin_root, + manifest=manifest, + origin=origin, + enabled=enabled, + ) + ) + return _demote_unsatisfied(plugins, errors) + + +def _demote_unsatisfied( + plugins: list[LoadedPlugin], errors: list[PluginLoadError] +) -> PluginLoadResult: + """Disable plugins whose declared dependencies are not present+enabled. + + A dependency is matched by name (discovery de-dups by name). Demotions are + recorded as per-plugin load errors so ``/doctor``-style surfaces can explain + why a plugin contributes nothing. + """ + from pythinker_code.plugin.dependency import verify_and_demote + + enabled_names = {p.name for p in plugins if p.enabled} + names_with_deps = [(p.name, p.manifest.dependencies) for p in plugins] + demoted, issues = verify_and_demote(names_with_deps, enabled_names) + if not demoted: + return PluginLoadResult(plugins=plugins, errors=errors) + + roots = {p.name: p.root for p in plugins} + plugins = [replace(p, enabled=False) if p.name in demoted else p for p in plugins] + errors = [ + *errors, + *(PluginLoadError(root=roots[issue.plugin], message=issue.message()) for issue in issues), + ] + return PluginLoadResult(plugins=plugins, errors=errors) diff --git a/src/pythinker_code/plugin/manifest.py b/src/pythinker_code/plugin/manifest.py new file mode 100644 index 00000000..345c2103 --- /dev/null +++ b/src/pythinker_code/plugin/manifest.py @@ -0,0 +1,260 @@ +"""Plugin and marketplace manifest schemas (Claude/Codex/pythinker compatible). + +A *plugin* is a directory that contributes reusable artifacts — skills, slash +commands, agents, hooks, MCP servers — to the agent. Its manifest may live in +``.pythinker-plugin/``, ``.claude-plugin/``, or ``.codex-plugin/`` (or at the +plugin root), all named ``plugin.json``. The schema is a superset of the three +ecosystems; unknown fields are ignored so forward-compatible manifests still load. + +This module is pure data + validation. Discovery, loading, and artifact +extraction live in :mod:`pythinker_code.plugin.loader` and +:mod:`pythinker_code.plugin.artifacts`. The legacy subprocess-tool ``PluginSpec`` +in :mod:`pythinker_code.plugin` is unrelated and stays as-is. +""" + +from __future__ import annotations + +import json +import re +from pathlib import Path +from typing import Any, cast + +from pydantic import BaseModel, ConfigDict, Field, field_validator + +# Manifest directory names tried in priority order, then the plugin root itself. +MANIFEST_DIRS: tuple[str, ...] = (".pythinker-plugin", ".claude-plugin", ".codex-plugin") +PLUGIN_MANIFEST = "plugin.json" +MARKETPLACE_MANIFEST = "marketplace.json" + + +class PluginManifestError(Exception): + """Raised when a plugin/marketplace manifest is missing or malformed.""" + + +class Author(BaseModel): + """Plugin/marketplace author. Accepts a bare string or an object.""" + + model_config = ConfigDict(extra="ignore") + + name: str = "" + email: str | None = None + url: str | None = None + + +def _coerce_author(value: Any) -> Any: + """Allow ``author`` to be a plain string (treated as the name).""" + if isinstance(value, str): + return {"name": value} + return value + + +def _as_str_list(value: Any) -> list[str]: + """Normalize a ``str | list[str] | None`` artifact path field to a list. + + The object-map form (``{name: {source: ...}}``) some Claude manifests use is + not modeled here; convention-dir fallback in the loader still finds those + artifacts, so we drop the explicit override rather than fail. + ponytail: add object-map parsing only when a real plugin needs it. + """ + if value is None: + return [] + if isinstance(value, str): + return [value] + if isinstance(value, list): + return [item for item in cast("list[object]", value) if isinstance(item, str)] + return [] + + +# A trailing ``@^`` segment is a forward-compat version constraint we do +# not yet enforce; strip it so the dep is matched by name[@marketplace] only. +_DEP_VERSION_SUFFIX = re.compile(r"@\^[^@]*$") + + +def _normalize_dependencies(value: Any) -> list[str]: + """Normalize ``dependencies`` entries to ``"name"`` / ``"name@marketplace"``. + + Accepts the string form (``"name"``, ``"name@mkt"``, ``"name@mkt@^1.2"``) and + the object form (``{"name": ..., "marketplace": ...}``) used by some Claude + manifests. Version suffixes (``@^...``) are stripped; malformed entries drop. + """ + if not isinstance(value, list): + return [] + out: list[str] = [] + for entry in cast("list[object]", value): + if isinstance(entry, str): + # Trim surrounding whitespace: downstream matching is exact-name, so a + # stray " lib " would be mismatched and falsely demoted as missing. + dep = _DEP_VERSION_SUFFIX.sub("", entry).strip() + if dep: + out.append(dep) + elif isinstance(entry, dict): + entry_d = cast("dict[str, object]", entry) + name = entry_d.get("name") + if not isinstance(name, str): + continue + name = name.strip() + if not name: + continue + marketplace = entry_d.get("marketplace") + if isinstance(marketplace, str) and marketplace.strip(): + out.append(f"{name}@{marketplace.strip()}") + else: + out.append(name) + return out + + +class PluginManifest(BaseModel): + """Parsed ``plugin.json`` for an artifact-contributing plugin.""" + + model_config = ConfigDict(extra="ignore") + + name: str + version: str = "" + description: str = "" + author: Author | None = None + homepage: str | None = None + repository: str | None = None + license: str | None = None + keywords: list[str] = Field(default_factory=list) + + # Artifact path overrides (relative to the plugin root). An empty list means + # "use the convention directory" (skills/, commands/, agents/, hooks/). + commands: list[str] = Field(default_factory=list) + agents: list[str] = Field(default_factory=list) + skills: list[str] = Field(default_factory=list) + output_styles: list[str] = Field(default_factory=list, alias="outputStyles") + + # Hooks: inline object or path(s); resolved by artifacts.py. Kept raw and + # narrowed at use sites (object, not Any, so it stays fully typed). + hooks: object = None + # MCP servers: name -> server config (or a path/ref resolved later). + mcp_servers: dict[str, object] = Field(default_factory=dict, alias="mcpServers") + + # Plugin dependencies: "name" or "name@marketplace". + dependencies: list[str] = Field(default_factory=list) + + # User-configurable options ({key: {type, title, sensitive, ...}}). Kept raw; + # only its presence + keys are used today (values are substituted into MCP/hook + # artifacts). Typed validation lands with the enable-time prompt. + user_config: dict[str, object] = Field(default_factory=dict, alias="userConfig") + + @field_validator("author", mode="before") + @classmethod + def _author(cls, v: Any) -> Any: + return _coerce_author(v) + + @field_validator("commands", "agents", "skills", "output_styles", mode="before") + @classmethod + def _paths(cls, v: Any) -> list[str]: + return _as_str_list(v) + + @field_validator("dependencies", mode="before") + @classmethod + def _dependencies(cls, v: Any) -> list[str]: + return _normalize_dependencies(v) + + +class MarketplaceEntry(BaseModel): + """One plugin listed in a marketplace manifest.""" + + model_config = ConfigDict(extra="ignore") + + name: str + version: str = "" + description: str = "" + author: Author | None = None + # Plugin source: a relative path string ("./plugins/foo") or a source object. + source: object = None + category: str | None = None + tags: list[str] = Field(default_factory=list) + strict: bool = True + + @field_validator("author", mode="before") + @classmethod + def _author(cls, v: Any) -> Any: + return _coerce_author(v) + + +class MarketplaceMetadata(BaseModel): + model_config = ConfigDict(extra="ignore") + + version: str = "" + description: str = "" + plugin_root: str | None = Field(default=None, alias="pluginRoot") + + +class MarketplaceManifest(BaseModel): + """Parsed ``marketplace.json``: a curated catalog of plugins.""" + + model_config = ConfigDict(extra="ignore") + + name: str + owner: Author | None = None + metadata: MarketplaceMetadata = Field(default_factory=MarketplaceMetadata) + # pyright strict flags list-of-model + default_factory as partially unknown; + # same pydantic pattern handled at plugin/__init__.py PluginSpec.tools. + plugins: list[MarketplaceEntry] = Field(default_factory=list) # pyright: ignore[reportUnknownVariableType] + + @field_validator("owner", mode="before") + @classmethod + def _owner(cls, v: Any) -> Any: + return _coerce_author(v) + + +def find_plugin_manifest(plugin_root: Path) -> Path | None: + """Return the path to a plugin's ``plugin.json``, or None if absent. + + Tries ``.pythinker-plugin/``, ``.claude-plugin/``, ``.codex-plugin/`` then the + plugin root, so plugins authored for any of the three ecosystems load. + """ + for manifest_dir in MANIFEST_DIRS: + candidate = plugin_root / manifest_dir / PLUGIN_MANIFEST + if candidate.is_file(): + return candidate + root_manifest = plugin_root / PLUGIN_MANIFEST + return root_manifest if root_manifest.is_file() else None + + +def _load_json(path: Path) -> dict[str, Any]: + try: + data = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as exc: + raise PluginManifestError(f"Failed to read {path}: {exc}") from exc + if not isinstance(data, dict): + raise PluginManifestError(f"Manifest must be a JSON object: {path}") + return cast("dict[str, Any]", data) + + +def load_plugin_manifest(plugin_root: Path) -> PluginManifest: + """Locate and parse a plugin's manifest. Raises :class:`PluginManifestError`.""" + manifest_path = find_plugin_manifest(plugin_root) + if manifest_path is None: + raise PluginManifestError(f"No {PLUGIN_MANIFEST} found under {plugin_root}") + data = _load_json(manifest_path) + if "name" not in data: + raise PluginManifestError(f"Missing required field 'name' in {manifest_path}") + try: + return PluginManifest.model_validate(data) + except Exception as exc: # pydantic ValidationError -> typed manifest error + raise PluginManifestError(f"Invalid plugin manifest {manifest_path}: {exc}") from exc + + +def find_marketplace_manifest(root: Path) -> Path | None: + """Return the path to a marketplace's ``marketplace.json``, or None.""" + for manifest_dir in MANIFEST_DIRS: + candidate = root / manifest_dir / MARKETPLACE_MANIFEST + if candidate.is_file(): + return candidate + root_manifest = root / MARKETPLACE_MANIFEST + return root_manifest if root_manifest.is_file() else None + + +def load_marketplace_manifest(path: Path) -> MarketplaceManifest: + """Parse a marketplace manifest from an exact file path.""" + data = _load_json(path) + if "name" not in data: + raise PluginManifestError(f"Missing required field 'name' in {path}") + try: + return MarketplaceManifest.model_validate(data) + except Exception as exc: + raise PluginManifestError(f"Invalid marketplace manifest {path}: {exc}") from exc diff --git a/src/pythinker_code/plugin/marketplace.py b/src/pythinker_code/plugin/marketplace.py new file mode 100644 index 00000000..6f7419b8 --- /dev/null +++ b/src/pythinker_code/plugin/marketplace.py @@ -0,0 +1,208 @@ +"""Marketplace registry: state, source parsing, and local resolution. + +Mirrors the reference (``blackbox/pythinker-src`` ``utils/plugins``): a +*marketplace* is a named catalog of plugins. Configured marketplaces are tracked +in ``known_marketplaces.json`` as ``{name: {source, installLocation, +lastUpdated, autoUpdate}}``; each ``source`` is a discriminated union +(github/git/url/file/directory/npm). + +State I/O is fail-soft (a corrupt entry is skipped, never crashes discovery) and +writes are atomic (temp + ``os.replace``) so a concurrent reader never sees a +torn file. Network/git fetching lives in a separate phase; this module covers +state and local (file/directory) sources. +""" + +from __future__ import annotations + +import json +import os +from pathlib import Path +from typing import Any, Literal, cast + +from pydantic import BaseModel, ConfigDict, Field, ValidationError + +from pythinker_code.plugin.directories import known_marketplaces_file, marketplaces_cache_dir +from pythinker_code.plugin.manifest import ( + MarketplaceManifest, + find_marketplace_manifest, + load_marketplace_manifest, +) +from pythinker_code.utils.io import file_lock +from pythinker_code.utils.logging import logger + +MarketplaceSourceKind = Literal["github", "git", "url", "file", "directory", "npm"] + + +class MarketplaceError(Exception): + """Raised for an invalid marketplace source or unresolvable marketplace.""" + + +class MarketplaceSource(BaseModel): + """Where a marketplace's manifest comes from.""" + + model_config = ConfigDict(extra="ignore") + + source: MarketplaceSourceKind + repo: str | None = None # github "owner/repo" + url: str | None = None # git/url + path: str | None = None # file/directory (absolute) + ref: str | None = None # git ref / branch + package: str | None = None # npm + + +class KnownMarketplaceEntry(BaseModel): + """One configured marketplace in ``known_marketplaces.json``.""" + + model_config = ConfigDict(extra="ignore", populate_by_name=True) + + source: MarketplaceSource + install_location: str | None = Field(default=None, alias="installLocation") + last_updated: str | None = Field(default=None, alias="lastUpdated") + auto_update: bool = Field(default=False, alias="autoUpdate") + + +def parse_marketplace_input(raw: str) -> MarketplaceSource: + """Parse a user-supplied marketplace string into a typed source. + + Mirrors the reference rules: SSH/git URLs, ``github.com`` URLs and + ``owner/repo`` shorthand → git/github; ``.git`` or ``/_git/`` → git; other + http(s) → url; local ``.json`` file → file; local dir → directory. Raises + :class:`MarketplaceError` for a missing path or an unrecognized input. + """ + trimmed = raw.strip() + if not trimmed: + raise MarketplaceError("Empty marketplace source") + + # SSH form: user@host:path(.git)?(#ref)? + import re + + ssh = re.match(r"^([a-zA-Z0-9._-]+@[^:]+:.+?(?:\.git)?)(#(.+))?$", trimmed) + if ssh: + return MarketplaceSource(source="git", url=ssh.group(1), ref=ssh.group(3)) + + if trimmed.startswith("http://"): + # Plaintext HTTP is tamperable in transit and can steer plugin + # install/update decisions; require HTTPS for remote marketplaces. + raise MarketplaceError(f"Insecure http:// marketplace source; use https://: {trimmed}") + if trimmed.startswith("https://"): + frag = re.match(r"^([^#]+)(#(.+))?$", trimmed) + url = frag.group(1) if frag else trimmed + ref = frag.group(3) if frag else None + if url.endswith(".git") or "/_git/" in url: + return MarketplaceSource(source="git", url=url, ref=ref) + if re.match(r"^https://(www\.)?github\.com/[^/]+/[^/]+", url): + git_url = url if url.endswith(".git") else f"{url}.git" + return MarketplaceSource(source="git", url=git_url, ref=ref) + return MarketplaceSource(source="url", url=url, ref=ref) + + if trimmed.startswith(("./", "../", "/", "~")): + resolved = Path(trimmed).expanduser().resolve() + if not resolved.exists(): + raise MarketplaceError(f"Path does not exist: {resolved}") + if resolved.is_file(): + if resolved.suffix != ".json": + raise MarketplaceError(f"Marketplace file must be .json: {resolved}") + return MarketplaceSource(source="file", path=str(resolved)) + if resolved.is_dir(): + return MarketplaceSource(source="directory", path=str(resolved)) + raise MarketplaceError(f"Cannot use path: {resolved}") + + # Shorthand owner/repo[(#|@)ref] -> github + if "/" in trimmed and not trimmed.startswith("@"): + if ":" in trimmed: + raise MarketplaceError(f"Unrecognized marketplace source: {trimmed}") + m = re.match(r"^([^#@]+)(?:[#@](.+))?$", trimmed) + repo = m.group(1) if m else trimmed + ref = m.group(2) if m else None + return MarketplaceSource(source="github", repo=repo, ref=ref) + + raise MarketplaceError(f"Unrecognized marketplace source: {trimmed}") + + +def _atomic_write_json(path: Path, data: Any) -> None: + """Write JSON atomically (temp file + ``os.replace``).""" + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_suffix(path.suffix + ".tmp") + tmp.write_text(json.dumps(data, ensure_ascii=False, indent=2), encoding="utf-8") + os.replace(tmp, path) + + +def load_known_marketplaces() -> dict[str, KnownMarketplaceEntry]: + """Load configured marketplaces, skipping any malformed entry (fail-soft).""" + path = known_marketplaces_file() + if not path.is_file(): + return {} + try: + raw = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as exc: + logger.warning("Cannot read known_marketplaces.json: {error}", error=exc) + return {} + if not isinstance(raw, dict): + return {} + result: dict[str, KnownMarketplaceEntry] = {} + for name, entry in cast("dict[str, Any]", raw).items(): + try: + result[name] = KnownMarketplaceEntry.model_validate(entry) + except ValidationError as exc: + logger.warning("Skipping invalid marketplace '{name}': {error}", name=name, error=exc) + return result + + +def save_known_marketplaces(marketplaces: dict[str, KnownMarketplaceEntry]) -> None: + """Persist the marketplace registry atomically.""" + payload = { + name: entry.model_dump(by_alias=True, exclude_none=True) + for name, entry in marketplaces.items() + } + _atomic_write_json(known_marketplaces_file(), payload) + + +def add_marketplace(name: str, source: MarketplaceSource, *, auto_update: bool = False) -> None: + """Register (or replace) a marketplace by name. + + The load → mutate → save runs under a cross-process lock so concurrent CLI + invocations can't drop each other's registry updates. + """ + with file_lock(known_marketplaces_file()): + marketplaces = load_known_marketplaces() + install_location = str(marketplaces_cache_dir() / name) + marketplaces[name] = KnownMarketplaceEntry( + source=source, + installLocation=install_location, + autoUpdate=auto_update, + ) + save_known_marketplaces(marketplaces) + + +def remove_marketplace(name: str) -> bool: + """Unregister a marketplace. Returns True if it existed. + + The read-modify-write runs under the same cross-process lock as + :func:`add_marketplace`. + """ + with file_lock(known_marketplaces_file()): + marketplaces = load_known_marketplaces() + if name not in marketplaces: + return False + del marketplaces[name] + save_known_marketplaces(marketplaces) + return True + + +def resolve_local_marketplace(source: MarketplaceSource) -> MarketplaceManifest: + """Load a marketplace manifest from a local file/directory source. + + git/url sources require fetching (a later phase); calling this on them raises. + """ + if source.source == "file": + if not source.path: + raise MarketplaceError("file marketplace source missing path") + return load_marketplace_manifest(Path(source.path)) + if source.source == "directory": + if not source.path: + raise MarketplaceError("directory marketplace source missing path") + manifest_path = find_marketplace_manifest(Path(source.path)) + if manifest_path is None: + raise MarketplaceError(f"No marketplace.json under {source.path}") + return load_marketplace_manifest(manifest_path) + raise MarketplaceError(f"Source '{source.source}' requires fetching, not local resolution") diff --git a/src/pythinker_code/plugin/options.py b/src/pythinker_code/plugin/options.py new file mode 100644 index 00000000..3b4e7224 --- /dev/null +++ b/src/pythinker_code/plugin/options.py @@ -0,0 +1,45 @@ +"""Plugin user-config value substitution. + +Plugins may declare ``userConfig`` options whose values the user fills in (via +``[plugins.options.]`` in config) and references as ``${user_config.KEY}`` +in executable artifacts — MCP server configs and hook commands. Substitution +mirrors the reference: a referenced key with no value is an error, so the caller +skips that artifact rather than running it with a silent blank. + +Deferred (each needs changes outside the plugin package): + * substitution in skill/agent/command *content* — those artifacts don't carry + their originating plugin id, so the value source can't be resolved there yet; + * the ``PYTHINKER_PLUGIN_OPTION_*`` hook environment variables — ``HookDef`` has + no ``env`` field; + * keychain-backed storage for ``sensitive`` options — values come from config + today; + * the interactive enable-time prompt + typed-schema validation. +See docs/en/customization/plugins.md. +""" + +from __future__ import annotations + +import re + +_USER_CONFIG_VAR = re.compile(r"\$\{user_config\.([^}]+)\}") + + +class UserConfigError(KeyError): + """A ``${user_config.KEY}`` referenced a value that is not configured.""" + + +def substitute_user_config_vars(text: str, values: dict[str, object]) -> str: + """Replace ``${user_config.KEY}`` in *text* with the configured value. + + Values are coerced with ``str()``. Raises :class:`UserConfigError` on the + first key with no configured value — callers skip the affected artifact + instead of running it with a blank. + """ + + def _replace(match: re.Match[str]) -> str: + key = match.group(1) + if key not in values: + raise UserConfigError(key) + return str(values[key]) + + return _USER_CONFIG_VAR.sub(_replace, text) diff --git a/src/pythinker_code/plugin/policy.py b/src/pythinker_code/plugin/policy.py new file mode 100644 index 00000000..66f94cd4 --- /dev/null +++ b/src/pythinker_code/plugin/policy.py @@ -0,0 +1,87 @@ +"""Session-scoped plugin activation policy. + +The policy decides whether external (Claude/Codex) plugins are activated and +which plugins are enabled. It is set once at startup from config and read by the +artifact collectors in :mod:`pythinker_code.plugin.integration`. A ``ContextVar`` +(not a plain global) carries it, mirroring the session-id ContextVar pattern, so +it propagates to subagent tasks and is safe under concurrency. +""" + +from __future__ import annotations + +from contextvars import ContextVar, Token +from dataclasses import dataclass, field + + +def _empty_options() -> dict[str, dict[str, object]]: + return {} + + +@dataclass(frozen=True) +class PluginPolicy: + """Which plugins a session activates. + + External (Claude/Codex) plugins are auto-detected by default for their *safe* + artifacts — skills, commands, agents — which are model-invoked, never + auto-run. Their *executable* artifacts — hooks and MCP servers — auto-run, so + they stay opt-in behind ``external_exec``. + """ + + discover_external: bool = True + external_exec: bool = False + # None enables all discovered plugins; a frozenset enables only those named. + enabled: frozenset[str] | None = None + # Names explicitly turned off; excluded even when ``enabled`` would allow them. + # This is how "disable" works under auto-detect (all-on) defaults. + disabled: frozenset[str] = frozenset() + # Per-plugin user-config values ({plugin_name: {option_key: value}}), filled + # into ``${user_config.KEY}`` references in MCP/hook artifacts. + options: dict[str, dict[str, object]] = field(default_factory=_empty_options) + + +_DEFAULT_POLICY = PluginPolicy() +_current_policy: ContextVar[PluginPolicy] = ContextVar("plugin_policy", default=_DEFAULT_POLICY) + + +def current_plugin_policy() -> PluginPolicy: + """The active plugin policy. + + Defaults to auto-detecting external plugins' safe artifacts + (``discover_external=True``), executable artifacts opt-in + (``external_exec=False``), and all discovered plugins enabled. + """ + return _current_policy.get() + + +def set_plugin_policy(policy: PluginPolicy) -> Token[PluginPolicy]: + """Install *policy* for the current context; returns a token for reset.""" + return _current_policy.set(policy) + + +def reset_plugin_policy(token: Token[PluginPolicy]) -> None: + """Restore the policy replaced by :func:`set_plugin_policy`.""" + _current_policy.reset(token) + + +def policy_from_config( + discover_external: bool, + external_exec: bool, + enabled: list[str], + disabled: list[str] | None = None, + options: dict[str, dict[str, object]] | None = None, +) -> PluginPolicy: + """Build a :class:`PluginPolicy` from config values. + + Blank/whitespace ``enabled`` entries are dropped so a stray ``[""]`` cannot + silently disable every plugin: an empty or all-blank list means "enable all" + (``None``), never "enable none". Blanks in ``disabled`` are dropped too. + """ + names = frozenset(name.strip() for name in enabled if name.strip()) + off = frozenset(name.strip() for name in (disabled or []) if name.strip()) + return PluginPolicy( + discover_external=discover_external, + external_exec=external_exec, + enabled=names or None, + disabled=off, + options=options or {}, + ) diff --git a/src/pythinker_code/prompt_templates.py b/src/pythinker_code/prompt_templates.py index 9b0de20f..f8917393 100644 --- a/src/pythinker_code/prompt_templates.py +++ b/src/pythinker_code/prompt_templates.py @@ -9,7 +9,7 @@ from pythinker_code.utils.logging import logger from pythinker_code.utils.path import find_project_root -PromptTemplateScope = Literal["project", "user"] +PromptTemplateScope = Literal["project", "user", "plugin"] @dataclass(frozen=True, slots=True) @@ -90,6 +90,8 @@ async def discover_prompt_templates(work_dir: HostPath) -> dict[str, PromptTempl Project templates win over user templates. """ + from pythinker_code.plugin.integration import plugin_command_dirs + roots: list[tuple[PromptTemplateScope, HostPath]] = [] project_root = await find_project_root(work_dir) roots.extend( @@ -98,6 +100,8 @@ async def discover_prompt_templates(work_dir: HostPath) -> dict[str, PromptTempl ("user", HostPath.home() / ".pythinker" / "prompts"), ] ) + # Enabled plugins contribute commands at lowest priority (project/user win). + roots.extend(("plugin", HostPath.unsafe_from_local_path(d)) for d in plugin_command_dirs()) templates: dict[str, PromptTemplate] = {} for scope, root in roots: diff --git a/src/pythinker_code/skill/__init__.py b/src/pythinker_code/skill/__init__.py index cf8a602c..9ec08453 100644 --- a/src/pythinker_code/skill/__init__.py +++ b/src/pythinker_code/skill/__init__.py @@ -208,7 +208,7 @@ async def resolve_skills_roots( Non-existent entries are silently dropped. Duplicates collapse to one. """ - from pythinker_code.plugin.manager import get_plugins_dir + from pythinker_code.plugin.integration import plugin_skill_dirs from pythinker_code.utils.path import find_project_root scoped: list[ScopedSkillsRoot] = [] @@ -275,13 +275,11 @@ def _append(root: HostPath, scope: SkillScope) -> None: # Plugins are always discoverable; treat as "extra" origin for prompt # grouping but place them below config-declared extras (user intent wins). - plugins_path = get_plugins_dir() - try: - plugins_is_dir = plugins_path.is_dir() - except OSError: - plugins_is_dir = False - if plugins_is_dir: - _append(HostPath.unsafe_from_local_path(plugins_path), "extra") + # Each enabled plugin contributes its own ``skills/`` root (resolved across + # pythinker/Claude/Codex installs), so skills nested inside a plugin — the + # common layout — are found, not just top-level dirs under the plugins root. + for skills_root in plugin_skill_dirs(): + _append(HostPath.unsafe_from_local_path(skills_root), "extra") if _supports_builtin_skills(): _append( diff --git a/src/pythinker_code/soul/agent.py b/src/pythinker_code/soul/agent.py index 0d2c38e5..da029c35 100644 --- a/src/pythinker_code/soul/agent.py +++ b/src/pythinker_code/soul/agent.py @@ -614,24 +614,49 @@ async def load_agent( continue toolset.add(plugin_tool) - if mcp_configs: - validated_mcp_configs: list[MCPConfig] = [] - if mcp_configs: - from fastmcp.mcp_config import MCPConfig + # Plugin-contributed MCP servers spawn processes, so load them only for the + # root agent (subagent_id is None) — never once per subagent. + plugin_mcp: dict[str, object] = {} + if runtime.subagent_id is None: + from pythinker_code.plugin.integration import plugin_mcp_servers - for mcp_config in mcp_configs: - try: - validated_mcp_configs.append( - mcp_config - if isinstance(mcp_config, MCPConfig) - else MCPConfig.model_validate(mcp_config) - ) - except pydantic.ValidationError as e: - raise MCPConfigError(f"Invalid MCP config: {e}") from e - if start_mcp_loading: - await toolset.load_mcp_tools(validated_mcp_configs, runtime, in_background=True) - else: - toolset.defer_mcp_tool_loading(validated_mcp_configs, runtime) + plugin_mcp = plugin_mcp_servers() + + if mcp_configs or plugin_mcp: + from fastmcp.mcp_config import MCPConfig + + from pythinker_code.cli.mcp import prepare_mcp_config_dict + + validated_mcp_configs: list[MCPConfig] = [] + for mcp_config in mcp_configs: + try: + raw_config = ( + mcp_config + if isinstance(mcp_config, dict) + else mcp_config.model_dump(mode="json") + ) + prepare_mcp_config_dict(raw_config) + validated_mcp_configs.append(MCPConfig.model_validate(raw_config)) + except pydantic.ValidationError as e: + # User-provided config: fail loud so the mistake is visible. + raise MCPConfigError(f"Invalid MCP config: {e}") from e + if plugin_mcp: + # Plugin config: fail soft — a malformed plugin MCP block is skipped + # with a warning rather than aborting the whole agent load. + try: + plugin_raw: dict[str, Any] = {"mcpServers": plugin_mcp} + prepare_mcp_config_dict(plugin_raw) + validated_mcp_configs.append(MCPConfig.model_validate(plugin_raw)) + except (pydantic.ValidationError, ValueError, TypeError, KeyError, AttributeError) as e: + # Fail soft: normalization (prepare_mcp_config_dict) can raise + # shape errors beyond ValidationError; a malformed plugin MCP + # block is skipped with a warning, never aborts agent load. + logger.warning("Skipping invalid plugin MCP servers: {error}", error=e) + if validated_mcp_configs: + if start_mcp_loading: + await toolset.load_mcp_tools(validated_mcp_configs, runtime, in_background=True) + else: + toolset.defer_mcp_tool_loading(validated_mcp_configs, runtime) return Agent( name=agent_spec.name, diff --git a/src/pythinker_code/soul/dynamic_injections/git_status.py b/src/pythinker_code/soul/dynamic_injections/git_status.py new file mode 100644 index 00000000..827b2093 --- /dev/null +++ b/src/pythinker_code/soul/dynamic_injections/git_status.py @@ -0,0 +1,63 @@ +from __future__ import annotations + +import hashlib +from collections.abc import Sequence +from typing import TYPE_CHECKING + +from pythinker_core.message import Message + +from pythinker_code.soul.dynamic_injection import DynamicInjection, DynamicInjectionProvider +from pythinker_code.subagents.git_context import collect_git_context + +if TYPE_CHECKING: + from pythinker_code.soul.pythinkersoul import PythinkerSoul + +_INJECTION_TYPE = "git_status" +_STALE_PREAMBLE = ( + "Git snapshot (point-in-time, may be stale by the time you act — re-run " + "`git status` / `git diff` before relying on it for irreversible decisions):\n" +) + + +class GitStatusInjectionProvider(DynamicInjectionProvider): + """Root-only injection of a bounded git working-tree snapshot. + + Reuses :func:`collect_git_context` so collection stays consistent with + read-oriented subagents. Re-injects when the snapshot changes or after compaction. + """ + + def __init__(self) -> None: + self._last_fingerprint: str | None = None + + async def get_injections( + self, + history: Sequence[Message], + soul: PythinkerSoul, + ) -> list[DynamicInjection]: + if soul.is_subagent or not soul.runtime.config.git_status_injection: + return [] + + raw = await collect_git_context(soul.runtime.work_dir) + if not raw: + return [] + + fingerprint = hashlib.sha256(raw.encode("utf-8")).hexdigest() + if fingerprint == self._last_fingerprint: + return [] + self._last_fingerprint = fingerprint + + inner = raw + prefix = "\n" + suffix = "\n" + if inner.startswith(prefix) and inner.endswith(suffix): + inner = inner[len(prefix) : -len(suffix)] + + return [ + DynamicInjection( + type=_INJECTION_TYPE, + content=_STALE_PREAMBLE + inner, + ) + ] + + async def on_context_compacted(self) -> None: + self._last_fingerprint = None diff --git a/src/pythinker_code/soul/live_tokens.py b/src/pythinker_code/soul/live_tokens.py new file mode 100644 index 00000000..39c679c7 --- /dev/null +++ b/src/pythinker_code/soul/live_tokens.py @@ -0,0 +1,47 @@ +"""Session-wide live output-token accumulator. + +Ports the reference design (``bootstrap/state.ts``: ``getTotalOutputTokens`` / +``getTurnOutputTokens`` / ``snapshotOutputTokensForTurn``): a single global total +that every in-process soul — root, subagents, background — increments as it +completes LLM steps. The live spinner reads the per-turn delta so the "↓ N tokens" +readout keeps moving during streaming, tool calls, and subagent runs instead of +freezing on the per-step context-size snapshot. + +ponytail: plain module-level ints — every soul shares one asyncio event loop, so +there is no cross-thread race to guard against. +""" + +from __future__ import annotations + +_total_output_tokens: int = 0 +_output_tokens_at_turn_start: int = 0 + + +def add_total_output_tokens(count: int) -> None: + """Add a completed step's output tokens to the session total.""" + global _total_output_tokens + if count > 0: + _total_output_tokens += count + + +def get_total_output_tokens() -> int: + """Session-cumulative output tokens across all in-process agents.""" + return _total_output_tokens + + +def snapshot_output_tokens_for_turn() -> None: + """Mark the start of a root turn so the live readout shows this turn's delta.""" + global _output_tokens_at_turn_start + _output_tokens_at_turn_start = _total_output_tokens + + +def get_turn_output_tokens() -> int: + """Output tokens produced since the current root turn began.""" + return max(0, _total_output_tokens - _output_tokens_at_turn_start) + + +def reset_for_tests() -> None: + """Reset module state (test isolation only).""" + global _total_output_tokens, _output_tokens_at_turn_start + _total_output_tokens = 0 + _output_tokens_at_turn_start = 0 diff --git a/src/pythinker_code/soul/pythinkersoul.py b/src/pythinker_code/soul/pythinkersoul.py index 604f1b55..f2bae3f7 100644 --- a/src/pythinker_code/soul/pythinkersoul.py +++ b/src/pythinker_code/soul/pythinkersoul.py @@ -89,6 +89,7 @@ ) from pythinker_code.soul.dynamic_injections.agent_list import AgentListInjectionProvider from pythinker_code.soul.dynamic_injections.auto_mode import AutoModeInjectionProvider +from pythinker_code.soul.dynamic_injections.git_status import GitStatusInjectionProvider from pythinker_code.soul.dynamic_injections.goal_mode import GoalModeInjectionProvider from pythinker_code.soul.dynamic_injections.inline_commands import InlineCommandReminderProvider from pythinker_code.soul.dynamic_injections.model_defense import ModelDefenseInjectionProvider @@ -96,6 +97,7 @@ from pythinker_code.soul.dynamic_injections.permissions_state import PermissionsInjectionProvider from pythinker_code.soul.dynamic_injections.plan_mode import PlanModeInjectionProvider from pythinker_code.soul.flow_runner import FLOW_COMMAND_PREFIX, FlowRunner +from pythinker_code.soul.live_tokens import add_total_output_tokens from pythinker_code.soul.message import ( check_message, system, @@ -198,7 +200,9 @@ def _is_hard_usage_limit(exception: BaseException) -> bool: return "usage_limit_reached" in text or "usage limit" in text -type StepStopReason = Literal["no_tool_calls", "tool_rejected", "stuck", "budget_exhausted"] +type StepStopReason = Literal[ + "no_tool_calls", "tool_rejected", "stuck", "budget_exhausted", "compaction_failed" +] _MISSING_REQUIRED_FIELD_RE = re.compile( @@ -287,6 +291,16 @@ def _budget_exhausted_message(session_cost_usd: float, ceiling: float) -> Messag return Message(role="assistant", content=[TextPart(text=text)]) +def _compaction_failed_message(failures: int, threshold: int) -> Message: + """Handoff message when proactive compaction cannot make progress.""" + text = ( + "Stopping: proactive context compaction failed " + f"{failures} consecutive time(s), reaching the configured threshold " + f"of {threshold}. I am handing back instead of retrying compaction blindly." + ) + return Message(role="assistant", content=[TextPart(text=text)]) + + def _crossed_budget_nudge_threshold( *, before_usd: float, @@ -499,6 +513,7 @@ def __init__( ) self._current_step_no = 0 self._consecutive_failures = 0 + self._compaction_failures = 0 self._truncation_recoveries = 0 # Cumulative LLM token usage for this soul instance (one run), so a subagent # can report its spend back to the orchestrating parent (subagent-2). A @@ -550,6 +565,8 @@ def __init__( # Self-filtering: root-only; posture-fingerprinted so it re-emits # exactly when yolo/auto/safe-mode/profile/session-approvals change. PermissionsInjectionProvider(), + # Self-filtering: root-only; bounded git snapshot for working-tree orientation. + GitStatusInjectionProvider(), # Self-filtering: root-only; keeps the model's subagent list current # without tying it to the static tool description cache. AgentListInjectionProvider(), @@ -1019,6 +1036,33 @@ async def wait_for_background_mcp_loading(self) -> None: return await self._agent.toolset.wait_for_mcp_tools() + async def disconnect_mcp_server(self, server_name: str) -> None: + if not isinstance(self._agent.toolset, PythinkerToolset): + return + await self._agent.toolset.disconnect_mcp_server(server_name, self._runtime) + wire_send(StatusUpdate(mcp_status=self._mcp_status_snapshot())) + + async def refresh_mcp_server(self, server_name: str) -> None: + if not isinstance(self._agent.toolset, PythinkerToolset): + return + await self._agent.toolset.refresh_mcp_server(server_name, self._runtime) + wire_send(StatusUpdate(mcp_status=self._mcp_status_snapshot())) + + async def reconnect_mcp_server(self, server_name: str) -> None: + if not isinstance(self._agent.toolset, PythinkerToolset): + return + await self._agent.toolset.reconnect_mcp_server(server_name, self._runtime) + wire_send(StatusUpdate(mcp_status=self._mcp_status_snapshot())) + + async def refresh_mcp_inventory(self, server_name: str | None = None) -> list[str]: + if not isinstance(self._agent.toolset, PythinkerToolset): + return [] + toolset = self._agent.toolset + targets = [server_name] if server_name else list(toolset.connected_mcp_server_names()) + for name in targets: + await self.refresh_mcp_server(name) + return targets + async def _checkpoint(self): await self._context.checkpoint(self._checkpoint_with_user_message) @@ -1611,9 +1655,11 @@ async def _agent_loop(self) -> TurnOutcome: logger.info("Context too long, compacting...") try: await self.compact_context() + self._compaction_failures = 0 except Exception as compact_err: from pythinker_code.telemetry.errors import report_handled_error + self._compaction_failures += 1 report_handled_error(compact_err, site="soul.context.compact") logger.error( "Context compaction failed at step {step_no}: {error_type}: {error}", @@ -1621,7 +1667,19 @@ async def _agent_loop(self) -> TurnOutcome: error_type=type(compact_err).__name__, error=compact_err, ) - raise + threshold = self._loop_control.max_compaction_failures + if self._compaction_failures >= threshold: + message = _compaction_failed_message( + self._compaction_failures, threshold + ) + await self._context.append_message(message) + wire_send(TextPart(text=message.extract_text(" "))) + return TurnOutcome( + stop_reason="compaction_failed", + final_message=message, + step_count=step_no - 1, + ) + # Below threshold: skip compaction this step and continue the turn. # Compaction makes a billable LLM call that folds into # self._session_cost_usd. Re-check the ceiling here so a session @@ -1637,6 +1695,8 @@ async def _agent_loop(self) -> TurnOutcome: final_message=message, step_count=step_no - 1, # this step's _step() never ran ) + else: + self._compaction_failures = 0 logger.debug("Beginning step {step_no}", step_no=step_no) await self._checkpoint() @@ -1873,6 +1933,10 @@ async def _run_step_once() -> StepResult: if u is not None: self._cumulative_usage = accumulate_usage(self._cumulative_usage, u) self._session_cost_usd += estimate_cost_usd(u, self.model_name) + # Feed the session-wide live counter so the spinner's "↓ tokens" + # readout reflects this step's output — including subagent and + # background souls, which all funnel through this one point. + add_total_output_tokens(u.output) def _opt_int(attr: str) -> int | None: """Read an optional usage counter as int — None when usage or the diff --git a/src/pythinker_code/soul/toolset.py b/src/pythinker_code/soul/toolset.py index 9411524a..2907acd5 100644 --- a/src/pythinker_code/soul/toolset.py +++ b/src/pythinker_code/soul/toolset.py @@ -2,6 +2,7 @@ import asyncio import contextlib +import copy import difflib import hashlib import importlib @@ -37,6 +38,7 @@ from pythinker_code.exception import InvalidToolError, MCPRuntimeError from pythinker_code.hooks.engine import HookEngine +from pythinker_code.telemetry.names import sanitize_telemetry_tool_name from pythinker_code.tools import SkipThisTool from pythinker_code.utils.logging import logger from pythinker_code.wire.types import ( @@ -236,6 +238,78 @@ def _set_transport_log_file(transport: Any) -> None: _set_transport_log_file(getattr(client, "transport", None)) +def _make_mcp_live_refresh_handler( + client: Any, + toolset: PythinkerToolset, + runtime: Runtime, + server_name: str, +) -> Any: + """Message handler that refreshes inventory on MCP list_changed notifications.""" + from fastmcp.client.tasks import TaskNotificationHandler + + class _McpLiveRefreshHandler(TaskNotificationHandler): + async def on_tool_list_changed( + self, message: mcp.types.ToolListChangedNotification + ) -> None: + await self._refresh_inventory("tools") + + async def on_resource_list_changed( + self, message: mcp.types.ResourceListChangedNotification + ) -> None: + await self._refresh_inventory("resources") + + async def on_prompt_list_changed( + self, message: mcp.types.PromptListChangedNotification + ) -> None: + await self._refresh_inventory("prompts") + + async def _refresh_inventory(self, capability: str) -> None: + info = toolset.mcp_servers.get(server_name) + if info is None or info.status != "connected": + return + try: + await toolset.refresh_mcp_server(server_name, runtime) + except Exception as exc: + logger.warning( + "MCP server {server_name} live {capability} refresh failed: {error}", + server_name=server_name, + capability=capability, + error=exc, + ) + + return _McpLiveRefreshHandler(client) + + +def _configure_mcp_client_handlers( + client: Any, + toolset: PythinkerToolset, + runtime: Runtime, + server_name: str, +) -> None: + _configure_mcp_client_stderr_log(client, runtime, server_name) + client._session_kwargs["message_handler"] = _make_mcp_live_refresh_handler( + client, toolset, runtime, server_name + ) + + +async def _hold_mcp_session(server_name: str, info: MCPServerInfo) -> None: + """Keep one MCP client session open so list_changed notifications can arrive.""" + stop = asyncio.Event() + info.session_stop = stop + try: + async with info.client: + await stop.wait() + except Exception as exc: + logger.debug( + "MCP session holder exited for {server_name}: {error}", + server_name=server_name, + error=exc, + ) + finally: + info.session_stop = None + info.session_holder_task = None + + def _classify_mcp_connect_error(error: BaseException, server_name: str) -> str: """One short actionable line for /mcp explaining a connect failure. @@ -534,6 +608,39 @@ def _register_mcp_tools(self, server_name: str, tools: list[MCPTool[Any]]) -> No ) self.add(tool) + def _publish_connected_mcp_tools(self, runtime: Runtime) -> None: + """Publish connected MCP tools in configured server order. + + Servers connect concurrently, so registering inside each connection task + makes duplicate tool-name resolution depend on task completion order. + Publishing after the gather keeps the collision policy deterministic. + """ + for server_name, server_info in self._mcp_servers.items(): + if server_info.status != "connected": + continue + self._register_mcp_tools(server_name, server_info.tools) + for tool in server_info.tools: + from pythinker_code.utils.mcp_names import mcp_tool_runtime_key + + runtime.mcp_tools[mcp_tool_runtime_key(server_name, tool.name)] = tool + + def _rebuild_published_mcp_tools(self, runtime: Runtime) -> None: + """Atomically rebuild the published MCP tool registry from connected servers. + + Drop every currently-published MCP tool (non-MCP tools are preserved), then + republish all *connected* servers in configured order via + :meth:`_publish_connected_mcp_tools`. This keeps last-wins collision order + deterministic and lets a disconnect/refresh re-claim a tool name another + still-connected server provides, instead of orphaning it. The method runs + synchronously (no ``await`` between the drop and the republish), so the two + registries are never observed half-rebuilt. + """ + stale = [name for name, tool in self._tool_dict.items() if isinstance(tool, MCPTool)] + for name in stale: + del self._tool_dict[name] + runtime.mcp_tools.clear() + self._publish_connected_mcp_tools(runtime) + def hide(self, tool_name: str) -> bool: """Hide a tool from the LLM tool list. Returns True if the tool exists.""" if tool_name in self._tool_dict: @@ -800,7 +907,7 @@ async def _call(): _current_tool_execution_started_ids.reset(started_ids_token) async def _call_with_lifecycle(): - tool_input_dict = arguments if isinstance(arguments, dict) else {} + tool_input_dict = copy.deepcopy(arguments) if isinstance(arguments, dict) else {} if self._runtime is not None: from pythinker_code.soul.permission import check_tool_call_allowed @@ -829,7 +936,7 @@ async def _call_with_lifecycle(): session_id=_get_session_id(), cwd=str(Path.cwd()), tool_name=tool_call.function.name, - tool_input=tool_input_dict, + tool_input=copy.deepcopy(tool_input_dict), tool_call_id=tool_call.id, ), ) @@ -857,19 +964,20 @@ async def _call_with_lifecycle(): emit_current_tool_execution_started() t0 = time.monotonic() + telemetry_tool_name = sanitize_telemetry_tool_name(tool_call.function.name) _tool_span_cm = _otel.start_span( "pythinker.tool", { - "tool.name": tool_call.function.name, + "tool.name": telemetry_tool_name, "tool.call_id": tool_call.id, # GenAI semconv so GenAI-aware backends recognize the tool layer. "gen_ai.operation.name": "execute_tool", - "gen_ai.tool.name": tool_call.function.name, + "gen_ai.tool.name": telemetry_tool_name, }, ) _tool_span = _tool_span_cm.__enter__() try: - ret = await self._gated_call(tool, arguments) + ret = await self._gated_call(tool, copy.deepcopy(arguments)) except Exception as e: tool_elapsed = time.monotonic() - t0 _tool_span.set_attribute("tool.success", False) @@ -877,7 +985,7 @@ async def _call_with_lifecycle(): _tool_span.set_attribute("tool.duration_ms", int(tool_elapsed * 1000)) _tool_span_cm.__exit__(type(e), e, e.__traceback__) _m.record_tool_call( - tool_name=tool_call.function.name, + tool_name=telemetry_tool_name, duration_seconds=tool_elapsed, success=False, error_type=type(e).__name__, @@ -896,7 +1004,7 @@ async def _call_with_lifecycle(): session_id=_get_session_id(), cwd=str(Path.cwd()), tool_name=tool_call.function.name, - tool_input=tool_input_dict, + tool_input=copy.deepcopy(tool_input_dict), error=str(e), tool_call_id=tool_call.id, ), @@ -906,12 +1014,12 @@ async def _call_with_lifecycle(): _error_type = type(e).__name__ track( "tool_error", - tool_name=tool_call.function.name, + tool_name=telemetry_tool_name, error_type=_error_type, ) track( "tool_call", - tool_name=tool_call.function.name, + tool_name=telemetry_tool_name, success=False, duration_ms=int(tool_elapsed * 1000), error_type=_error_type, @@ -936,7 +1044,7 @@ async def _call_with_lifecycle(): _tool_span.set_attribute("tool.duration_ms", int(tool_elapsed * 1000)) _tool_span_cm.__exit__(None, None, None) _m.record_tool_call( - tool_name=tool_call.function.name, + tool_name=telemetry_tool_name, duration_seconds=tool_elapsed, success=_tool_succeeded, ) @@ -950,7 +1058,7 @@ async def _call_with_lifecycle(): _track_tool_call( "tool_call", - tool_name=tool_call.function.name, + tool_name=telemetry_tool_name, success=not isinstance(ret, ToolError), duration_ms=int(tool_elapsed * 1000), dup_type="cross_step" if is_cross_step_dup else "normal", @@ -964,7 +1072,7 @@ async def _call_with_lifecycle(): session_id=_get_session_id(), cwd=str(Path.cwd()), tool_name=tool_call.function.name, - tool_input=tool_input_dict, + tool_input=copy.deepcopy(tool_input_dict), tool_output=str(ret)[:2000], tool_call_id=tool_call.id, ), @@ -1182,78 +1290,8 @@ async def _connect_server( ) -> tuple[str, Exception | None]: if server_info.status != "pending": return server_name, None - server_info.status = "connecting" - - async def _open_and_inventory() -> None: - async with server_info.client as client: - skipped: list[str] = [] - local_tools: list[MCPTool[Any]] = [] - for tool in await client.list_tools(): - if server_info.tool_filter and not server_info.tool_filter.allows( - tool.name - ): - skipped.append(tool.name) - continue - local_tools.append( - MCPTool( - server_name, - tool, - client, - runtime=runtime, - tool_filter=server_info.tool_filter, - ) - ) - if skipped: - logger.info( - "MCP server {server_name}: {n} tools filtered out by " - "mcp.json enabledTools/disabledTools: {names}", - server_name=server_name, - n=len(skipped), - names=", ".join(sorted(skipped)), - ) - # Resources/prompts are optional MCP capabilities; a server - # that exposes none (or does not support the request) must - # still connect, so capture them best-effort (mcpext-1). A - # METHOD_NOT_FOUND means the capability is genuinely absent; any - # other error is surfaced (WARNING) rather than masked as "none". - local_resources = await _discover_optional_capability( - server_name, "resources", client.list_resources - ) - local_prompts = await _discover_optional_capability( - server_name, "prompts", client.list_prompts - ) - server_info.tools = local_tools - server_info.resources = local_resources - server_info.prompts = local_prompts - - try: - # Bound connect+inventory: a hung server would otherwise block - # every agent turn (the loop awaits MCP loading). - await asyncio.wait_for( - _open_and_inventory(), - timeout=runtime.config.mcp.client.startup_timeout_ms / 1000, - ) - - self._register_mcp_tools(server_name, server_info.tools) - for tool in server_info.tools: - runtime.mcp_tools[f"mcp__{server_name}__{tool.name}"] = tool - - server_info.status = "connected" - logger.info("Connected MCP server: {server_name}", server_name=server_name) - return server_name, None - except Exception as e: - from pythinker_code.telemetry.errors import report_handled_error - - report_handled_error(e, site="soul.toolset.mcp.connect") - logger.error( - "Failed to connect MCP server: {server_name}, error: {error}", - server_name=server_name, - error=e, - ) - server_info.status = "failed" - server_info.error = _classify_mcp_connect_error(e, server_name) - return server_name, e + return await self._connect_mcp_server(server_name, server_info, runtime) async def _connect(): _toast_mcp("connecting to mcp servers...") @@ -1279,6 +1317,9 @@ async def _connect(): results = await asyncio.gather(*tasks) if tasks else [] failed_servers = {name: error for name, error in results if error is not None} + # Publish before raising so servers that DID connect become callable in + # this session even when another server fails the aggregate connect. + self._publish_connected_mcp_tools(runtime) if failed_servers: _toast_mcp("mcp connection failed") raise MCPRuntimeError(f"Failed to connect MCP servers: {failed_servers}") @@ -1297,7 +1338,7 @@ async def _connect(): oauth_servers[server_name] = server_config.url client = fastmcp.Client(MCPConfig(mcpServers={server_name: server_config})) - _configure_mcp_client_stderr_log(client, runtime, server_name) + _configure_mcp_client_handlers(client, self, runtime, server_name) self._mcp_servers[server_name] = MCPServerInfo( status="pending", client=client, @@ -1305,6 +1346,7 @@ async def _connect(): resources=[], prompts=[], tool_filter=McpToolFilter.from_server_config(server_config), + server_config=server_config, ) if not any(server_info.status == "pending" for server_info in self._mcp_servers.values()): @@ -1330,6 +1372,192 @@ async def wait_for_mcp_tools(self) -> None: if self._mcp_loading_task is task and task.done(): self._mcp_loading_task = None + async def _inventory_mcp_server( + self, server_name: str, server_info: MCPServerInfo, runtime: Runtime + ) -> tuple[list[MCPTool[Any]], list[mcp.Resource], list[mcp.types.Prompt]]: + """Discover a server's tools/resources/prompts without mutating it. + + Returns the freshly discovered inventory; the caller assigns it onto + ``server_info`` only after the awaited call (and the client context exit) + fully succeeds, so a timeout or ``__aexit__`` failure never leaves the + published registry inconsistent with the exposed callable tools. + """ + async with server_info.client as client: + skipped: list[str] = [] + local_tools: list[MCPTool[Any]] = [] + for tool in await client.list_tools(): + if server_info.tool_filter and not server_info.tool_filter.allows(tool.name): + skipped.append(tool.name) + continue + local_tools.append( + MCPTool( + server_name, + tool, + client, + runtime=runtime, + tool_filter=server_info.tool_filter, + ) + ) + if skipped: + logger.info( + "MCP server {server_name}: {n} tools filtered out by " + "mcp.json enabledTools/disabledTools: {names}", + server_name=server_name, + n=len(skipped), + names=", ".join(sorted(skipped)), + ) + resources = await _discover_optional_capability( + server_name, "resources", client.list_resources + ) + prompts = await _discover_optional_capability( + server_name, "prompts", client.list_prompts + ) + return local_tools, resources, prompts + + async def _connect_mcp_server( + self, server_name: str, server_info: MCPServerInfo, runtime: Runtime + ) -> tuple[str, Exception | None]: + try: + tools, resources, prompts = await asyncio.wait_for( + self._inventory_mcp_server(server_name, server_info, runtime), + timeout=runtime.config.mcp.client.startup_timeout_ms / 1000, + ) + # Assign only after the awaited inventory (and client context exit) + # succeeded, so a failure never leaves a half-applied inventory. + server_info.tools = tools + server_info.resources = resources + server_info.prompts = prompts + server_info.status = "connected" + server_info.error = None + self._start_mcp_session_holder(server_name, server_info) + logger.info("Connected MCP server: {server_name}", server_name=server_name) + return server_name, None + except Exception as e: + from pythinker_code.telemetry.errors import report_handled_error + + report_handled_error(e, site="soul.toolset.mcp.connect") + logger.error( + "Failed to connect MCP server: {server_name}, error: {error}", + server_name=server_name, + error=e, + ) + server_info.status = "failed" + server_info.error = _classify_mcp_connect_error(e, server_name) + return server_name, e + + def _ensure_mcp_idle(self) -> None: + if self._mcp_loading_task is not None and not self._mcp_loading_task.done(): + raise MCPRuntimeError("MCP servers are still loading") + + def _start_mcp_session_holder(self, server_name: str, info: MCPServerInfo) -> None: + task = info.session_holder_task + if task is not None and not task.done(): + return + info.session_holder_task = asyncio.create_task(_hold_mcp_session(server_name, info)) + + async def _stop_mcp_session_holder(self, info: MCPServerInfo) -> None: + if info.session_stop is not None: + info.session_stop.set() + task = info.session_holder_task + if task is None: + return + with contextlib.suppress(asyncio.CancelledError): + await task + + async def disconnect_mcp_server(self, server_name: str, runtime: Runtime) -> None: + """Disconnect one MCP server and unregister its tools.""" + self._ensure_mcp_idle() + info = self._mcp_servers.get(server_name) + if info is None: + raise MCPRuntimeError(f"Unknown MCP server: {server_name}") + await self._stop_mcp_session_holder(info) + # Drop this server's inventory, then rebuild the published registry so any + # tool name it was shadowing falls back to another still-connected server. + info.tools = [] + self._rebuild_published_mcp_tools(runtime) + prior_error = info.error + close_error: str | None = None + try: + await asyncio.wait_for(info.client.close(), timeout=_MCP_CLOSE_TIMEOUT_S) + except TimeoutError as exc: + logger.warning( + "MCP disconnect close timed out for {server_name}: {error}", + server_name=server_name, + error=exc, + ) + close_error = "disconnect timed out while closing the MCP session" + except Exception as exc: + logger.warning( + "MCP disconnect close failed for {server_name}: {error}", + server_name=server_name, + error=exc, + ) + close_error = f"disconnect failed while closing the MCP session: {exc}" + info.status = "failed" + if close_error is not None: + if prior_error is None: + info.error = close_error + else: + info.error = "disconnected" + info.resources = [] + info.prompts = [] + + def connected_mcp_server_names(self) -> tuple[str, ...]: + return tuple(name for name, info in self._mcp_servers.items() if info.status == "connected") + + async def refresh_mcp_server(self, server_name: str, runtime: Runtime) -> None: + """Re-list tools/resources/prompts for a connected MCP server.""" + self._ensure_mcp_idle() + info = self._mcp_servers.get(server_name) + if info is None: + raise MCPRuntimeError(f"Unknown MCP server: {server_name}") + if info.status != "connected": + raise MCPRuntimeError( + f"MCP server '{server_name}' is not connected (status={info.status})" + ) + # Inventory first: on failure ``info.tools`` keeps its last-known-good value + # and the live registry is untouched, so a failed refresh never drops tools. + # Convert raw timeout/inventory errors to MCPRuntimeError so callers (e.g. the + # /mcp slash handler) receive a single typed boundary error. + try: + tools, resources, prompts = await asyncio.wait_for( + self._inventory_mcp_server(server_name, info, runtime), + timeout=runtime.config.mcp.client.startup_timeout_ms / 1000, + ) + except TimeoutError as exc: + raise MCPRuntimeError(f"Refresh of MCP server '{server_name}' timed out") from exc + except Exception as exc: + raise MCPRuntimeError(f"Failed to refresh MCP server '{server_name}': {exc}") from exc + # Inventory succeeded; swap the old inventory for the new one atomically, + # then rebuild the published registry from it. + info.tools = tools + info.resources = resources + info.prompts = prompts + self._rebuild_published_mcp_tools(runtime) + + async def reconnect_mcp_server(self, server_name: str, runtime: Runtime) -> None: + """Close and reconnect one MCP server from its stored config.""" + import fastmcp + from fastmcp.mcp_config import MCPConfig + + self._ensure_mcp_idle() + info = self._mcp_servers.get(server_name) + if info is None: + raise MCPRuntimeError(f"Unknown MCP server: {server_name}") + if info.server_config is None: + raise MCPRuntimeError(f"MCP server '{server_name}' has no stored config to reconnect") + await self.disconnect_mcp_server(server_name, runtime) + info.client = fastmcp.Client(MCPConfig(mcpServers={server_name: info.server_config})) + _configure_mcp_client_handlers(info.client, self, runtime, server_name) + info.status = "pending" + info.error = None + _server_name, error = await self._connect_mcp_server(server_name, info, runtime) + if error is not None: + raise MCPRuntimeError( + info.error or f"Failed to reconnect MCP server '{server_name}': {error}" + ) + self._rebuild_published_mcp_tools(runtime) + async def cleanup(self) -> None: """Cleanup any resources held by the toolset.""" self._deferred_mcp_load = None @@ -1341,6 +1569,7 @@ async def cleanup(self) -> None: # Close every MCP client concurrently with a per-server timeout, so one # hung or slow client cannot block teardown of the rest (mcpext-3). async def _close(info: MCPServerInfo) -> None: + await self._stop_mcp_session_holder(info) try: await asyncio.wait_for(info.client.close(), timeout=_MCP_CLOSE_TIMEOUT_S) except Exception as exc: @@ -1362,6 +1591,11 @@ class MCPServerInfo: error: str | None = None # Optional mcp.json enabledTools/disabledTools scoping for this server. tool_filter: McpToolFilter | None = None + # Original server config for per-server reconnect (mcpext-2). + server_config: Any = None + # Background task holding the client session open for list_changed notifications. + session_stop: asyncio.Event | None = None + session_holder_task: asyncio.Task[None] | None = None class MCPTool[T: ClientTransport](CallableTool): @@ -1447,7 +1681,10 @@ async def __call__(self, *args: Any, **kwargs: Any) -> ToolReturnValue: return result.rejection_error() from pythinker_code.telemetry import otel as _otel + from pythinker_code.telemetry.names import sanitize_telemetry_tool_name + telemetry_server = sanitize_telemetry_tool_name(self._mcp_server_name) + telemetry_tool = sanitize_telemetry_tool_name(self._mcp_tool.name) # `start_span` returns a sync context manager (the OTel SDK uses # `_AgnosticContextManager`, which intentionally has no __aenter__). # Keep it as a sync `with` and use `async with` only on the fastmcp @@ -1456,11 +1693,11 @@ async def __call__(self, *args: Any, **kwargs: Any) -> ToolReturnValue: with _otel.start_span( "pythinker.mcp.call", { - "mcp.server": self._mcp_server_name, - "mcp.tool": self._mcp_tool.name, + "mcp.server": telemetry_server, + "mcp.tool": telemetry_tool, "mcp.timeout_ms": int(self._timeout.total_seconds() * 1000), "gen_ai.operation.name": "execute_tool", - "gen_ai.tool.name": self._mcp_tool.name, + "gen_ai.tool.name": telemetry_tool, }, ) as span: async with self._client as client: diff --git a/src/pythinker_code/subagents/discovery.py b/src/pythinker_code/subagents/discovery.py index 566fcbd6..bf16c855 100644 --- a/src/pythinker_code/subagents/discovery.py +++ b/src/pythinker_code/subagents/discovery.py @@ -16,7 +16,7 @@ from pythinker_code.utils.logging import logger from pythinker_code.utils.path import find_project_root -AgentScope = Literal["project"] +AgentScope = Literal["project", "plugin"] CLAUDE_TOOL_MAP: dict[str, str] = { "Agent": "pythinker_code.tools.agent:Agent", @@ -46,9 +46,11 @@ class MarkdownAgentSpec: prompt_file: HostPath scope: AgentScope tools: tuple[str, ...] | None = None + exclude_tools: tuple[str, ...] | None = None model: str | None = None when_to_use: str = "" required_mcp_servers: tuple[str, ...] = () + steps: int | None = None def _project_agent_dir_candidates(project_root: HostPath) -> tuple[HostPath, ...]: @@ -82,6 +84,13 @@ async def add_existing(candidates: Iterable[HostPath], scope: AgentScope) -> Non roots.append(ScopedAgentRoot(root=canon, scope=scope)) await add_existing(_project_agent_dir_candidates(project_root), "project") + + # Enabled plugins (pythinker/Claude/Codex installs) contribute agent roots + # below project scope, so a project-local agent of the same name wins. + from pythinker_code.plugin.integration import plugin_agent_dirs + + plugin_roots = [HostPath.unsafe_from_local_path(d) for d in plugin_agent_dirs()] + await add_existing(plugin_roots, "plugin") return roots @@ -129,6 +138,11 @@ def parse_markdown_agent( model = _as_nonempty_str(fm.get("model")) when_to_use = _as_nonempty_str(fm.get("when_to_use")) or description tools = _map_tools(fm.get("tools"), source=prompt_file) + exclude_source = fm.get("disallowed_tools") + if exclude_source is None: + exclude_source = fm.get("exclude_tools") + exclude_tools = _map_tools(exclude_source, source=prompt_file) + steps = _as_positive_int(fm.get("max_turns")) or _as_positive_int(fm.get("steps")) raw_required = fm.get("required_mcp_servers") if raw_required is not None and not isinstance(raw_required, list): logger.info( @@ -148,9 +162,11 @@ def parse_markdown_agent( prompt_file=prompt_file, scope=scope, tools=tools, + exclude_tools=exclude_tools, model=model, when_to_use=when_to_use, required_mcp_servers=required_mcp_servers, + steps=steps, ) @@ -209,8 +225,12 @@ def materialize_markdown_agent_specs( model = agent.model if available_models is None or agent.model in available_models else None if model: payload["agent"]["model"] = model + if agent.steps is not None: + payload["agent"]["steps"] = agent.steps if agent.tools is not None: payload["agent"]["allowed_tools"] = list(agent.tools) + if agent.exclude_tools is not None: + payload["agent"]["exclude_tools"] = list(agent.exclude_tools) wrapper_path.write_text(yaml.safe_dump(payload, sort_keys=False), encoding="utf-8") policy = ( ToolPolicy(mode="allowlist", tools=agent.tools) @@ -254,6 +274,17 @@ def _map_tools(value: Any, *, source: HostPath) -> tuple[str, ...] | None: return tuple(mapped) +def _as_positive_int(value: Any) -> int | None: + if isinstance(value, bool): + return None + if isinstance(value, int) and value >= 1: + return value + if isinstance(value, str) and value.strip().isdigit(): + parsed = int(value.strip()) + return parsed if parsed >= 1 else None + return None + + def _as_nonempty_str(value: Any) -> str | None: return value.strip() if isinstance(value, str) and value.strip() else None diff --git a/src/pythinker_code/telemetry/names.py b/src/pythinker_code/telemetry/names.py new file mode 100644 index 00000000..b488c2e1 --- /dev/null +++ b/src/pythinker_code/telemetry/names.py @@ -0,0 +1,44 @@ +"""Telemetry-safe tool name normalization (obs-eval-6). + +Span and metric attributes must not carry raw MCP server paths, secrets, or +unbounded plugin identifiers. Runtime tool names stay unchanged for the model; +only telemetry exports use these sanitized labels. +""" + +from __future__ import annotations + +import hashlib +import re + +_TELEMETRY_TOOL_NAME_MAX = 64 +_UNSAFE_CHARS = re.compile(r"[^A-Za-z0-9_.-]+") + + +def sanitize_telemetry_tool_name(name: str) -> str: + """Return a bounded, path-safe tool label for spans and metrics.""" + trimmed = name.strip() + if not trimmed: + return "unknown_tool" + if trimmed.startswith("mcp__"): + parts = trimmed.split("__", 2) + if len(parts) == 3: + _prefix, server, tool = parts + return _bounded_mcp_label(server, tool) + cleaned = _UNSAFE_CHARS.sub("_", trimmed).strip("_") + if not cleaned: + cleaned = f"tool_{hashlib.sha256(trimmed.encode('utf-8')).hexdigest()[:8]}" + return cleaned[:_TELEMETRY_TOOL_NAME_MAX] + + +def _bounded_mcp_label(server: str, tool: str) -> str: + # The server segment is user-named and can embed a path, account, hostname, + # or token-like value, so it is hashed — never exported raw — per the + # no-secrets/PII telemetry contract. The tool segment comes from the server's + # published tool list (not user input) and stays readable for analytics. + server_hash = hashlib.sha256(server.strip().encode("utf-8")).hexdigest()[:8] + safe_tool = _UNSAFE_CHARS.sub("_", tool).strip("_") or "tool" + label = f"mcp__{server_hash}__{safe_tool}" + if len(label) <= _TELEMETRY_TOOL_NAME_MAX: + return label + digest = hashlib.sha256(label.encode("utf-8")).hexdigest()[:8] + return f"{label[: _TELEMETRY_TOOL_NAME_MAX - 9]}_{digest}" diff --git a/src/pythinker_code/tools/file/read_media.py b/src/pythinker_code/tools/file/read_media.py index f109a6c5..926a773f 100644 --- a/src/pythinker_code/tools/file/read_media.py +++ b/src/pythinker_code/tools/file/read_media.py @@ -13,11 +13,17 @@ from pythinker_code.tools.file.utils import MEDIA_SNIFF_BYTES, FileType, detect_file_type from pythinker_code.tools.utils import load_desc from pythinker_code.utils.logging import logger +from pythinker_code.utils.media_limits import ( + MAX_IMAGE_BYTES, + MAX_IMAGE_PIXELS, + MAX_VIDEO_BYTES, + format_byte_limit, +) from pythinker_code.utils.media_tags import wrap_media_part from pythinker_code.utils.path import is_within_workspace from pythinker_code.wire.types import ImageURLPart, VideoURLPart -MAX_MEDIA_MEGABYTES = 100 +MAX_MEDIA_MEGABYTES = max(MAX_IMAGE_BYTES, MAX_VIDEO_BYTES) // (1024 * 1024) def _to_data_url(mime_type: str, data: bytes) -> str: @@ -101,11 +107,12 @@ async def _read_media(self, path: HostPath, file_type: FileType) -> ToolReturnVa message=f"`{path}` is empty.", brief="Empty file", ) - if size > (MAX_MEDIA_MEGABYTES << 20): + max_bytes = MAX_IMAGE_BYTES if file_type.kind == "image" else MAX_VIDEO_BYTES + if size > max_bytes: return ToolError( message=( f"`{path}` is {size} bytes, which exceeds the max " - f"{MAX_MEDIA_MEGABYTES}MB bytes for media files." + f"{format_byte_limit(max_bytes)} limit for {file_type.kind} files." ), brief="File too large", ) @@ -113,10 +120,20 @@ async def _read_media(self, path: HostPath, file_type: FileType) -> ToolReturnVa match file_type.kind: case "image": data = await path.read_bytes() + image_size = _extract_image_size(data) + if image_size is not None: + width, height = image_size + if width * height > MAX_IMAGE_PIXELS: + return ToolError( + message=( + f"`{path}` is {width}x{height}px, which exceeds the max " + f"{MAX_IMAGE_PIXELS:,} pixel limit for images." + ), + brief="Image too large", + ) data_url = _to_data_url(file_type.mime_type, data) part = ImageURLPart(image_url=ImageURLPart.ImageURL(url=data_url)) wrapped = wrap_media_part(part, tag="image", attrs={"path": media_path}) - image_size = _extract_image_size(data) case "video": data = await path.read_bytes() if (llm := self._runtime.llm) and isinstance(llm.chat_provider, Pythinker): diff --git a/src/pythinker_code/tools/mcp_resource/__init__.py b/src/pythinker_code/tools/mcp_resource/__init__.py index fd56a72f..7abfdc3c 100644 --- a/src/pythinker_code/tools/mcp_resource/__init__.py +++ b/src/pythinker_code/tools/mcp_resource/__init__.py @@ -8,7 +8,7 @@ """ from pathlib import Path -from typing import Any +from typing import Any, cast from pydantic import BaseModel, Field from pythinker_core.tooling import CallableTool2, ToolError, ToolOk, ToolReturnValue @@ -119,3 +119,80 @@ async def __call__(self, params: ReadParams) -> ToolReturnValue: # treats it as data, never instructions. builder.mark_untrusted() return builder.ok(f"Read resource {params.uri} from {params.server}.") + + +class PromptParams(BaseModel): + server: str = Field(description="The connected MCP server that publishes the prompt.") + name: str = Field(description="The prompt name from ListMcpResources.") + arguments: dict[str, Any] = Field( + default_factory=dict, + description="Structured prompt arguments to pass to the MCP server.", + ) + + +def _render_prompt_messages(messages: object) -> str: + if not isinstance(messages, list): + return "" + lines: list[str] = [] + for message_obj in cast(list[object], messages): + role = str(getattr(message_obj, "role", "unknown")) + content: object = getattr(message_obj, "content", "") + if isinstance(content, str): + rendered = content + else: + text = getattr(content, "text", None) + if isinstance(text, str): + rendered = text + else: + # Non-text content: emit a bounded placeholder rather than + # stringifying a possibly large/opaque object into model context + # (same safe handling as ReadMcpResource above). + blob: Any = getattr(content, "blob", None) + mime = getattr(content, "mimeType", None) or "application/octet-stream" + size = f"{len(blob)} bytes" if isinstance(blob, (bytes, str)) else "size unknown" + rendered = f"[binary content omitted: {mime}, {size}]" + lines.append(f"role: {role}\ncontent:\n{rendered}") + return "\n\n".join(lines).strip() + + +class InvokeMcpPrompt(CallableTool2[PromptParams]): + name: str = "InvokeMcpPrompt" + supports_parallel: bool = True + params: type[PromptParams] = PromptParams + + def __init__(self, toolset: PythinkerToolset) -> None: + super().__init__(description=load_desc(Path(__file__).parent / "prompt_description.md")) + self._toolset = toolset + + async def __call__(self, params: PromptParams) -> ToolReturnValue: + info = self._toolset.mcp_servers.get(params.server) + if info is None: + available = ", ".join(sorted(self._toolset.mcp_servers)) or "(none connected)" + return ToolError( + message=f"Unknown MCP server: {params.server}. Connected servers: {available}", + brief="Unknown MCP server", + ) + prompt_names = {prompt.name for prompt in info.prompts} + if params.name not in prompt_names: + available = ", ".join(sorted(prompt_names)) or "(none published)" + return ToolError( + message=( + f"Unknown MCP prompt: {params.name} on {params.server}. " + f"Published prompts: {available}" + ), + brief="Unknown MCP prompt", + ) + try: + async with info.client as client: + prompt_result = await client.get_prompt(params.name, params.arguments) + except Exception as exc: + return ToolError( + message=f"Failed to invoke prompt {params.name} from {params.server}: {exc}", + brief="Prompt invocation failed", + ) + + rendered = _render_prompt_messages(getattr(prompt_result, "messages", [])) + builder = ToolResultBuilder() + builder.write(rendered or "(prompt returned no messages)") + builder.mark_untrusted() + return builder.ok(f"Invoked prompt {params.name} from {params.server}.") diff --git a/src/pythinker_code/tools/mcp_resource/prompt_description.md b/src/pythinker_code/tools/mcp_resource/prompt_description.md new file mode 100644 index 00000000..9e0efc71 --- /dev/null +++ b/src/pythinker_code/tools/mcp_resource/prompt_description.md @@ -0,0 +1,17 @@ +# Invoke MCP Prompt + +Invoke a prompt template published by a connected MCP server. + +First discover the `server` and prompt `name` with ListMcpResources, then pass +any structured `arguments` required by the prompt. The returned prompt messages +are wrapped as untrusted data because they come from an external MCP server. + +When to use: +- After ListMcpResources shows a prompt template that would help with the task. +- When the user explicitly asks to use an MCP server's published prompt. + +When NOT to use: +- To call an MCP tool — those are already in your toolset; call them directly. +- To read an MCP resource — use ReadMcpResource with the resource URI. + +This is read-only and always available. diff --git a/src/pythinker_code/tools/plan/__init__.py b/src/pythinker_code/tools/plan/__init__.py index 61531708..76787987 100644 --- a/src/pythinker_code/tools/plan/__init__.py +++ b/src/pythinker_code/tools/plan/__init__.py @@ -4,6 +4,7 @@ import asyncio import logging +import re from collections.abc import Awaitable, Callable from pathlib import Path from typing import override @@ -29,6 +30,12 @@ _RESERVED_LABELS = {"reject", "revise", "approve", "reject and exit"} +_VERIFICATION_SECTION_RE = re.compile(r"^#+\s*verification\b", re.MULTILINE | re.IGNORECASE) + + +def _plan_lacks_verification_section(content: str) -> bool: + return _VERIFICATION_SECTION_RE.search(content) is None + class PlanOption(BaseModel): """A selectable approach/option within the plan.""" @@ -221,7 +228,14 @@ def build_handoff_output(selected_option: str | None = None) -> str: ] # Display plan content inline in the chat - wire_send(PlanDisplay(content=plan_content, file_path=str(plan_path))) + display_content = plan_content + if _plan_lacks_verification_section(plan_content): + display_content = ( + f"{plan_content}\n\n" + "> **Warning:** This plan has no Verification section. Add the smallest " + "command, test, or check per meaningful change before approval." + ) + wire_send(PlanDisplay(content=display_content, file_path=str(plan_path))) request = QuestionRequest( id=str(uuid4()), diff --git a/src/pythinker_code/tools/plan/description.md b/src/pythinker_code/tools/plan/description.md index e1334463..0c2a7358 100644 --- a/src/pythinker_code/tools/plan/description.md +++ b/src/pythinker_code/tools/plan/description.md @@ -20,6 +20,9 @@ If your plan contains multiple alternative approaches: ## Before Using - Yolo mode does not auto-approve this tool. In yolo mode, this tool still presents the plan to the user for approval. +- The plan file must include a Verification section. For each meaningful change, + name the smallest command, test, or check that would prove the change worked + end-to-end. - If auto mode is active, do NOT use AskUserQuestion; make the best decision from available context. - If auto mode is active, this tool is auto-approved because no user is present. - If auto mode is not active and you have unresolved questions, use AskUserQuestion first. diff --git a/src/pythinker_code/tools/recall/__init__.py b/src/pythinker_code/tools/recall/__init__.py index 69dcd949..0fbcddab 100644 --- a/src/pythinker_code/tools/recall/__init__.py +++ b/src/pythinker_code/tools/recall/__init__.py @@ -39,26 +39,52 @@ class Params(BaseModel): ) query: str | None = Field( default=None, - description="Keywords to match against prior session titles (mode=search). " - "Omit to list recent sessions.", + description=( + "Keywords to match against prior session titles, session_ids, and plan slugs " + "(mode=search). Omit to list recent sessions." + ), ) session_id: str | None = Field( default=None, description="The session_id to read, from a prior Recall search (mode=read).", ) + message_offset: int = Field( + default=0, + ge=0, + description=( + "For mode=read, skip this many rendered non-internal transcript messages before " + "returning content." + ), + ) + max_messages: int | None = Field( + default=None, + ge=1, + le=200, + description="For mode=read, return at most this many rendered transcript messages.", + ) + + +def _session_search_blob(session: Session) -> str: + """Lexical search text for a session (title, id, plan slug).""" + title = (session.state.custom_title or session.title or "").strip() + parts = [title, session.id] + plan_slug = session.state.plan_slug + if plan_slug: + parts.append(plan_slug) + return " ".join(parts).lower() def _rank_sessions( sessions: list[Session], *, query: str, current_id: str, limit: int ) -> list[Session]: - """Rank prior sessions by title keyword overlap then recency (pure).""" + """Rank prior sessions by keyword overlap then recency (pure).""" terms = query.lower().split() scored: list[tuple[int, float, Session]] = [] for session in sessions: if session.id == current_id: continue - title = (session.state.custom_title or session.title or "").strip() - score = sum(1 for term in terms if term in title.lower()) + blob = _session_search_blob(session) + score = sum(1 for term in terms if term in blob) if terms and score == 0: continue scored.append((score, session.updated_at, session)) @@ -66,15 +92,31 @@ def _rank_sessions( return [session for _score, _ts, session in scored[:limit]] -def _render_transcript(context_file: Path, budget: int) -> str: +def _render_transcript( + context_file: Path, + budget: int, + *, + message_offset: int = 0, + max_messages: int | None = None, +) -> str: """Render a session's message log into a budgeted, sanitized transcript. Internal (``_``-prefixed) roles are skipped. Each message's text is sanitized; a block that trips the secret/injection scanner becomes ``[redacted]`` rather - than leaking or silently vanishing. Stops once the char budget is reached. + than leaking or silently vanishing. + + Windowing operates on the renderable (post-filter) message stream — i.e. after + internal-role and empty-segment messages are dropped: + + - ``message_offset``: skip this many renderable messages before emitting any. + - ``max_messages``: emit at most this many renderable messages (``None`` = no limit). + + Stops once the char budget or ``max_messages`` is reached. """ out: list[str] = [] used = 0 + seen_messages = 0 + included_messages = 0 try: # Stream line-by-line (not read_text) so a huge transcript cannot blow up # memory; errors="replace" tolerates a corrupt/binary line without crashing. @@ -97,12 +139,19 @@ def _render_transcript(context_file: Path, budget: int) -> str: segment = f"{segment} [tool calls: {', '.join(tool_names)}]".strip() if not segment: continue + if seen_messages < message_offset: + seen_messages += 1 + continue + if max_messages is not None and included_messages >= max_messages: + break clean = sanitize_candidate_block(segment) entry = f"[{role}] {clean if clean is not None else '[redacted]'}" if used + len(entry) + 1 > budget: out.append("… (transcript truncated to fit the recall budget)") break out.append(entry) + seen_messages += 1 + included_messages += 1 used += len(entry) + 1 except OSError: return "" @@ -134,7 +183,11 @@ async def __call__(self, params: Params) -> ToolReturnValue: message=f"Invalid session_id: {session_id!r}.", brief="Invalid session_id", ) - return await self._read(session_id) + return await self._read( + session_id, + message_offset=params.message_offset, + max_messages=params.max_messages, + ) async def _search(self, query: str) -> ToolReturnValue: work_dir = self._runtime.work_dir @@ -159,11 +212,15 @@ async def _search(self, query: str) -> ToolReturnValue: title = session.state.custom_title or session.title or "(untitled)" lines.append(f"- session_id: {session.id}") lines.append(f" title: {title}") + if session.state.plan_slug: + lines.append(f" plan_slug: {session.state.plan_slug}") lines.append("") lines.append('Read one with Recall(mode="read", session_id="...").') return ToolOk(output="\n".join(lines), message=f"Found {len(top)} prior session(s).") - async def _read(self, session_id: str) -> ToolReturnValue: + async def _read( + self, session_id: str, *, message_offset: int, max_messages: int | None + ) -> ToolReturnValue: if session_id == self._runtime.session.id: return ToolError(message="Cannot recall the current session.", brief="Current session") work_dir = self._runtime.work_dir @@ -181,7 +238,11 @@ async def _read(self, session_id: str) -> ToolReturnValue: ) rendered = await asyncio.to_thread( - _render_transcript, session.context_file, _READ_BUDGET_CHARS + _render_transcript, + session.context_file, + _READ_BUDGET_CHARS, + message_offset=message_offset, + max_messages=max_messages, ) if not rendered: return ToolOk( diff --git a/src/pythinker_code/tools/recall/description.md b/src/pythinker_code/tools/recall/description.md index ca349e16..30d0ec83 100644 --- a/src/pythinker_code/tools/recall/description.md +++ b/src/pythinker_code/tools/recall/description.md @@ -5,10 +5,13 @@ commands, and file paths from an earlier session when you need to repeat or exte prior work. Two modes: -- `mode="search"` — find prior sessions by keyword over their titles. Pass `query` - (omit to list recent sessions). Returns session_ids + titles. +- `mode="search"` — find prior sessions by keyword over their titles, session_ids, + and plan slugs. Pass `query` (omit to list recent sessions). Returns + session_ids, titles, and plan slugs when present. - `mode="read"` — read a chosen session's transcript. Pass `session_id` (from a - prior search). Returns a budgeted, sanitized transcript. + prior search). Optionally pass `message_offset` and `max_messages` to read a + bounded window of rendered non-internal transcript messages. Returns a budgeted, + sanitized transcript. When to use: - The user references earlier work ("continue what we did on the auth migration"). diff --git a/src/pythinker_code/ui/shell/__init__.py b/src/pythinker_code/ui/shell/__init__.py index aedbdee0..8d248eac 100644 --- a/src/pythinker_code/ui/shell/__init__.py +++ b/src/pythinker_code/ui/shell/__init__.py @@ -2701,7 +2701,7 @@ def _panel() -> Panel: # Boot animation: blink the antenna 7 times, then stop. Only when the # Unicode logo actually rendered, on a real terminal tall enough that the # antenna row is still on screen, and never under reduced motion. - if logo_rendered and console.is_terminal and not motion_disabled(): + if logo_rendered and console.is_terminal and not console.record and not motion_disabled(): cell = _antenna_cell(panel, panel_width) if cell is not None: rows_up, column = cell diff --git a/src/pythinker_code/ui/shell/components/markdown.py b/src/pythinker_code/ui/shell/components/markdown.py index 4094fd2b..53c144f2 100644 --- a/src/pythinker_code/ui/shell/components/markdown.py +++ b/src/pythinker_code/ui/shell/components/markdown.py @@ -75,6 +75,7 @@ sorted(_MARKDOWN_ICON_REPLACEMENTS, key=len, reverse=True) ) _FENCE_RE = re.compile(r"^(?P {0,3})(?P`{3,}|~{3,})") +_OL_ITEM_RE = re.compile(r"^\d+\.\s") _PRIORITY_MATRIX_ROW_RE = re.compile( r"^\s*(?P[A-Z]{1,3}\d+)\s*(?:[─━—-]|\s){2,}\s*" r"(?PCRITICAL|HIGH|MEDIUM|LOW|INFO)\s*$", @@ -718,6 +719,44 @@ def _normalize_table_block(text: str) -> str: text = remainder if remainder.startswith("\n") else "\n" + remainder +def _loosen_tight_ordered_lists(markup: str) -> str: + """Insert a blank line before each ordered-list item that immediately follows another.""" + lines = markup.splitlines(keepends=True) + out: list[str] = [] + in_fence = False + fence_char = "" + fence_len = 0 + prev_was_ol = False + + for line in lines: + m = _FENCE_RE.match(line) + if in_fence: + out.append(line) + if m: + fence = m.group("fence") + if fence.startswith(fence_char) and len(fence) >= fence_len: + in_fence = False + fence_char = "" + fence_len = 0 + prev_was_ol = False + continue + if m: + in_fence = True + fence_char = m.group("fence")[0] + fence_len = len(m.group("fence")) + out.append(line) + prev_was_ol = False + continue + + is_ol = bool(_OL_ITEM_RE.match(line)) + if is_ol and prev_was_ol: + out.append("\n") + out.append(line) + prev_was_ol = is_ol + + return "".join(out) + + def _normalize_markdown_tables(markup: str) -> str: """Apply :func:`_normalize_table_block` to every fence-free span of markup.""" if "|" not in markup or "-" not in markup: @@ -779,7 +818,8 @@ def __init__(self, markup: str, *args: Any, **kwargs: Any) -> None: unwrapped_markup = _unwrap_fenced_markdown_tables(safe_markup) repaired_markup = _repair_crammed_markdown_tables(unwrapped_markup) normalized_markup = _normalize_markdown_tables(repaired_markup) - super().__init__(_simplify_markdown_report_icons(normalized_markup), *args, **kwargs) + loosened_markup = _loosen_tight_ordered_lists(normalized_markup) + super().__init__(_simplify_markdown_report_icons(loosened_markup), *args, **kwargs) def __rich_console__(self, console: Console, options: ConsoleOptions) -> RenderResult: overrides = _markdown_style_overrides() diff --git a/src/pythinker_code/ui/shell/components/report.py b/src/pythinker_code/ui/shell/components/report.py index 07bec644..48050793 100644 --- a/src/pythinker_code/ui/shell/components/report.py +++ b/src/pythinker_code/ui/shell/components/report.py @@ -31,6 +31,7 @@ from rich.panel import Panel from rich.rule import Rule from rich.style import Style as RichStyle +from rich.table import Table from rich.text import Text from pythinker_code.ui.shell.components.markdown import PythinkerMarkdown, pythinker_markdown @@ -302,9 +303,18 @@ def _summary_line(counts: dict[Severity, int], theme: ThemeName | None) -> Text: def _render_finding(finding: ReportFinding, theme: ThemeName | None) -> RenderableType: rows: list[RenderableType] = [] - title = Text() - title.append(f"{_DOT} ", style=_severity_style(finding.severity, theme)) - title.append(finding.title, style=tui_rich_style("border", theme=theme) + RichStyle(bold=True)) + # Hang-indent the title: the ● marker sits alone in a 2-wide gutter and the + # title text (and its wrapped lines) align at column 2 — the same column as + # the finding's location/body below. A flat ``Text`` wraps back under the + # marker, which flattened the hierarchy and made a wrapped title read like a + # new finding. + title = Table.grid(padding=0) + title.add_column(width=2, no_wrap=True) + title.add_column(overflow="fold") + title.add_row( + Text(_DOT, style=_severity_style(finding.severity, theme)), + Text(finding.title, style=tui_rich_style("border", theme=theme) + RichStyle(bold=True)), + ) rows.append(title) if finding.location: diff --git a/src/pythinker_code/ui/shell/prompt.py b/src/pythinker_code/ui/shell/prompt.py index 2c59fbd0..66c0133f 100644 --- a/src/pythinker_code/ui/shell/prompt.py +++ b/src/pythinker_code/ui/shell/prompt.py @@ -1589,8 +1589,9 @@ def _rank(c: Completion) -> tuple[int, ...]: cat = 1 else: cat = 2 + test_penalty = int(any("test" in segment.lower() for segment in path.split("/"))) # preserve original FuzzyCompleter's order in the same category - return (cat,) + return (cat, test_penalty) candidates.sort(key=_rank) yield from candidates @@ -2399,6 +2400,18 @@ def _(event: KeyPressEvent) -> None: track("shortcut_editor") self._open_in_external_editor(event) + def _has_staged_suggestion_prefill() -> bool: + return bool(getattr(self, "_staged_suggestion_prefill", None)) + + @_kb.add("escape", "s", eager=True, filter=Condition(_has_staged_suggestion_prefill)) + def _(event: KeyPressEvent) -> None: + """Accept the latest agent suggestion into the prompt buffer.""" + if self.accept_staged_suggestion_prefill(): + from pythinker_code.telemetry import track + + track("suggestion_accepted") + event.app.invalidate() + @_kb.add( "up", eager=True, @@ -3331,6 +3344,7 @@ def _render_background_working_status(self, columns: int) -> FormattedText: if total <= 0: # Background work drained — reset the elapsed/rate trackers. self._bg_status_started_at = None + self._bg_status_start_tokens = None samples = getattr(self, "_bg_token_samples", None) if samples is not None: samples.clear() @@ -3338,8 +3352,11 @@ def _render_background_working_status(self, columns: int) -> FormattedText: now = time.monotonic() started_at = getattr(self, "_bg_status_started_at", None) if started_at is None: + from pythinker_code.soul.live_tokens import get_total_output_tokens + started_at = now self._bg_status_started_at = now + self._bg_status_start_tokens = get_total_output_tokens() elapsed = max(0.0, now - started_at) frame = active_marker_frame(elapsed) tokens = _get_tui_tokens() @@ -3372,27 +3389,30 @@ def _background_status_metadata(self, now: float) -> str: """Compact ``(elapsed, ↓ Nk tokens, N t/s)`` suffix for the line above. Same visual language as the live view's working/todo headers. Elapsed - counts from when background work first appeared; the rate is a short - sliding window over the status snapshot's context tokens (mirroring - ``_ContentBlock._record_token_rate_sample``). + counts from when background work first appeared; the token readout is the + session-wide output tokens produced since this stretch began (detached + background souls feed the same counter), and the rate is a short sliding + window over it (mirroring ``_ContentBlock._record_token_rate_sample``). """ from pythinker_code.soul import format_token_count + from pythinker_code.soul.live_tokens import get_total_output_tokens from pythinker_code.utils.datetime import format_elapsed parts: list[str] = [] started = getattr(self, "_bg_status_started_at", None) if started is not None: parts.append(format_elapsed(max(0.0, now - started))) - provider = getattr(self, "_status_provider", None) - status = provider() if provider is not None else None - context_tokens = getattr(status, "context_tokens", None) or 0 - if context_tokens: - parts.append(f"↓ {format_token_count(context_tokens)} tokens") + start_tokens = getattr(self, "_bg_status_start_tokens", None) + bg_tokens = ( + max(0, get_total_output_tokens() - start_tokens) if start_tokens is not None else 0 + ) + if bg_tokens: + parts.append(f"↓ {format_token_count(bg_tokens)} tokens") samples: deque[tuple[float, int]] | None = getattr(self, "_bg_token_samples", None) if samples is None: samples = deque() self._bg_token_samples = samples - samples.append((now, context_tokens)) + samples.append((now, bg_tokens)) # 1.5s window, ≥3 samples — the live view's tracker parameters. while len(samples) > 1 and now - samples[0][0] > 1.5: samples.popleft() @@ -3563,6 +3583,23 @@ def set_prefill_text(self, text: str) -> None: """ self._prefill_text = text + def stage_suggestion_prefill(self, prefill: str) -> None: + """Remember a non-blocking Suggestion prefill until the user accepts it.""" + text = prefill.strip() + self._staged_suggestion_prefill = text or None + + def accept_staged_suggestion_prefill(self) -> bool: + """Insert a staged suggestion prefill into the prompt buffer.""" + text = getattr(self, "_staged_suggestion_prefill", None) + if not text: + return False + self._staged_suggestion_prefill = None + buffer = self._session.default_buffer + if buffer.text and not buffer.text.endswith((" ", "\n")): + buffer.insert_text(" ") + buffer.insert_text(text) + return True + async def prompt_next(self) -> UserInput: return await self._prompt_once(append_history=None) @@ -3642,6 +3679,7 @@ async def _prompt_once(self, *, append_history: bool | None) -> UserInput: # Consume one-shot prefill text if set default = getattr(self, "_prefill_text", None) or "" self._prefill_text = None + self._staged_suggestion_prefill = None with patch_stdout(raw=True): command = str( await self._session.prompt_async(placeholder=placeholder, default=default) diff --git a/src/pythinker_code/ui/shell/slash.py b/src/pythinker_code/ui/shell/slash.py index 2d2f4876..a0957150 100644 --- a/src/pythinker_code/ui/shell/slash.py +++ b/src/pythinker_code/ui/shell/slash.py @@ -2244,9 +2244,67 @@ async def _prompt_auto_update_selection(*, current: bool) -> bool | None: @registry.command async def mcp(app: Shell, args: str): - """Show MCP servers and tools""" + """Show MCP servers and tools, or manage one server (reconnect/disconnect/refresh).""" from rich.live import Live + from pythinker_code.exception import MCPRuntimeError + from pythinker_code.ui.theme import get_tui_tokens as _get_tok_mcp + + parts = args.strip().split() + if parts: + verb = parts[0].lower() + if verb in {"reconnect", "disconnect", "refresh", "retry"}: + soul = ensure_pythinker_soul(app) + if soul is None: + return + server_name = parts[1] if len(parts) > 1 else None + # Reject extra operands instead of silently using only the first. + if verb in {"reconnect", "disconnect", "retry"} and len(parts) != 2: + console.print(f"[{_get_tok_mcp().warning}]Usage: /mcp {verb} [/]") + return + if verb == "refresh" and len(parts) > 2: + console.print(f"[{_get_tok_mcp().warning}]Usage: /mcp refresh [server][/]") + return + if verb == "retry": + verb = "reconnect" + try: + if verb == "disconnect": + if not server_name: + console.print( + f"[{_get_tok_mcp().warning}]Usage: /mcp disconnect [/]" + ) + return + await soul.disconnect_mcp_server(server_name) + console.print( + f"[{_get_tok_mcp().success}]Disconnected MCP server " + f"{_rich_escape(server_name)}.[/]" + ) + return + if verb == "reconnect": + if not server_name: + console.print( + f"[{_get_tok_mcp().warning}]Usage: /mcp reconnect [/]" + ) + return + await soul.reconnect_mcp_server(server_name) + console.print( + f"[{_get_tok_mcp().success}]Reconnected MCP server " + f"{_rich_escape(server_name)}.[/]" + ) + return + # refresh [server] + targets = await soul.refresh_mcp_inventory(server_name) + if not targets: + console.print(f"[{_get_tok_mcp().warning}]No connected MCP servers.[/]") + return + console.print( + f"[{_get_tok_mcp().success}]Refreshed MCP inventory for " + f"{len(targets)} server(s).[/]" + ) + except MCPRuntimeError as exc: + console.print(f"[{_get_tok_mcp().warning}]{exc}[/]") + return + soul = ensure_pythinker_soul(app) if soul is None: return diff --git a/src/pythinker_code/ui/shell/tool_renderers/agent.py b/src/pythinker_code/ui/shell/tool_renderers/agent.py index 9846a3ce..6e3a800a 100644 --- a/src/pythinker_code/ui/shell/tool_renderers/agent.py +++ b/src/pythinker_code/ui/shell/tool_renderers/agent.py @@ -2,9 +2,10 @@ from __future__ import annotations +import json import re from dataclasses import dataclass -from typing import cast +from typing import TypedDict, cast from rich import box as rich_box from rich.console import Group, RenderableType @@ -60,6 +61,17 @@ _RE_SEVERITY_IN_HEADER = re.compile(r"\b(critical|high|medium|low)\b(?!-)", re.IGNORECASE) # Markdown table row starting with a severity cell: `| HIGH | description |` _RE_TABLE_SEVERITY_ROW = re.compile(r"^\|\s*(critical|high|medium|low)\s*\|", re.IGNORECASE) +# Primary machine-readable format: ```report\n{"findings":[{"severity":"high",...}]}\n``` +# ponytail: no line-start anchor — tolerates LLM fencing drift; tighten if reviewers stabilize +_RE_REPORT_BLOCK = re.compile(r"```report\s*\n(.*?)```", re.DOTALL) + + +class _ReportFinding(TypedDict, total=False): + severity: str + + +class _ReportBlock(TypedDict, total=False): + findings: list[_ReportFinding] @dataclass @@ -88,10 +100,55 @@ def _is_review_run(agents: list[dict[str, str]]) -> bool: def _parse_reviewer_findings(result_text: str) -> tuple[dict[str, int], bool]: """Parse severity counts from structured markers only (never mid-sentence prose). - Returns (severity_counts, was_parsed). was_parsed is True when at least one - structured marker was found; False means the whole report is unreadable prose. + Returns (severity_counts, was_parsed). was_parsed is True when a structured + report was found — including a valid ```report block with zero findings. + False means the whole report is unreadable prose with no structured markers. + + Primary: ```report JSON block ({"findings": [{"severity": "high", ...}, ...]}). + Any ```report block (even one with empty findings or malformed JSON) means this + is a structured report, so the markdown fallback is skipped entirely. + + Fallback: line-by-line markdown markers for prose-formatted reports. + The fallback only runs when no ```report block is present, so it never + re-scans JSON block content and cannot miscount severity words inside JSON. """ counts: dict[str, int] = {sev: 0 for sev in _SEVERITY_LABELS} + + # Primary: machine-readable ```report JSON block. Only a well-formed JSON + # object counts as "parsed" — a malformed block must not report success with + # zero findings; it falls through to the markdown fallback instead. + json_blocks = list(_RE_REPORT_BLOCK.finditer(result_text)) + if json_blocks: + parsed_valid = False + for block_match in json_blocks: + try: + parsed = json.loads(block_match.group(1)) + except json.JSONDecodeError: + continue # malformed JSON — block found but not parseable + if not isinstance(parsed, dict): + continue # wrong shape (array / scalar) + data = cast(_ReportBlock, parsed) + findings = data.get("findings") + if not isinstance(findings, list): + continue # "findings" missing or not a list — nothing to count + parsed_valid = True + # The declared type promises dict findings, but the payload is + # untrusted JSON; re-type as object so the runtime guard below is real. + for finding in cast("list[object]", findings): + if not isinstance(finding, dict): + continue # skip non-object finding entries + sev = str(cast("dict[str, object]", finding).get("severity", "")).lower() + if sev in counts: + counts[sev] += 1 + # A report block was present: return its counts. ``parsed_valid`` is True + # only when at least one block was a JSON object with a list ``findings`` + # (possibly empty); a malformed block or a wrong-shaped payload (e.g. + # ``{"findings": "high"}``) is reported as unparsed, never as a false + # "parsed with zero findings", and never re-scanned below. + return counts, parsed_valid + + # Fallback: line-by-line markdown markers. + # Only reached when no ```report block exists — result_text is JSON-free. found_any = False section_severity: str | None = None # set when inside e.g. "### High Severity" @@ -212,6 +269,13 @@ def _render_findings_table(summary: ReviewFindingsSummary) -> RenderableType: "low": "info", } + def _reported_by(names: list[str]) -> Text: + if not names: + return Text("—", style=tui_rich_style("dim")) + t = Text("— ", style=tui_rich_style("warning")) + t.append(", ".join(names)) + return t + for sev in _SEVERITY_LABELS: count = getattr(summary, sev) by = summary.reporters.get(sev, []) @@ -219,7 +283,7 @@ def _render_findings_table(summary: ReviewFindingsSummary) -> RenderableType: table.add_row( Text(sev.capitalize(), style=style), Text(str(count), style=style), - Text(", ".join(by) if by else "—", style=tui_rich_style("dim")), + _reported_by(by), ) if summary.unparsed_reports > 0: @@ -227,7 +291,7 @@ def _render_findings_table(summary: ReviewFindingsSummary) -> RenderableType: table.add_row( Text("Unknown", style=tui_rich_style("muted")), Text(str(summary.unparsed_reports), style=tui_rich_style("muted")), - Text(", ".join(by) if by else "—", style=tui_rich_style("dim")), + _reported_by(by), ) n, total = summary.parsed_reports, summary.total_reports diff --git a/src/pythinker_code/ui/shell/visualize/_blocks.py b/src/pythinker_code/ui/shell/visualize/_blocks.py index e36ea865..cde99920 100644 --- a/src/pythinker_code/ui/shell/visualize/_blocks.py +++ b/src/pythinker_code/ui/shell/visualize/_blocks.py @@ -1352,7 +1352,10 @@ def compose(self) -> RenderableType: label, bullet=Text(TRANSCRIPT_ASSISTANT_MARKER, style=tui_rich_style("accent")), ) - hint = Text(f"→ {prefill}", style=tui_rich_style("muted")) + hint = Text( + f"→ {prefill} (Alt+S to accept)", + style=tui_rich_style("muted"), + ) return BulletColumns( Group(label, hint), bullet=Text(TRANSCRIPT_ASSISTANT_MARKER, style=tui_rich_style("accent")), @@ -1412,10 +1415,16 @@ class _CompactionBlock: TIPS: tuple[str, ...] = FEATURE_TIPS - def __init__(self, *, context_tokens: int | None = None) -> None: + def __init__( + self, + *, + context_tokens: int | None = None, + todos_renderable: RenderableType | None = None, + ) -> None: self._start = time.monotonic() self._tip = random.choice(self.TIPS) self._context_tokens = context_tokens + self._todos_renderable = todos_renderable def update_context_tokens(self, context_tokens: int | None) -> None: """Refresh the token count shown in the compacting title.""" @@ -1449,7 +1458,9 @@ def _render(self) -> RenderableType: bar.append("▱" * empty, style=muted) bar.append(f" {pct}%", style=muted) + if self._todos_renderable is not None: + return Group(title, bar, self._todos_renderable) + tip = Text(" ⎿ ", style=muted) tip.append(f"Tip: {self._tip}", style=subtle) - return Group(title, bar, tip) diff --git a/src/pythinker_code/ui/shell/visualize/_interactive.py b/src/pythinker_code/ui/shell/visualize/_interactive.py index 7ca05a50..20017ecb 100644 --- a/src/pythinker_code/ui/shell/visualize/_interactive.py +++ b/src/pythinker_code/ui/shell/visualize/_interactive.py @@ -55,6 +55,7 @@ StatusUpdate, SteerInput, StepInterrupted, + Suggestion, TurnEnd, WireMessage, ) @@ -514,6 +515,13 @@ def dispatch_wire_message(self, msg: WireMessage) -> None: return super().dispatch_wire_message(msg) + def display_suggestion(self, event: Suggestion) -> None: + super().display_suggestion(event) + # Stage unconditionally: an empty prefill clears any prior staged value + # (stage_suggestion_prefill stores ``None`` for blank input), so a later + # suggestion without a prefill cannot leave stale Esc+s text behind. + self._prompt_session.stage_suggestion_prefill(event.prefill) + # -- Running prompt rendering -------------------------------------------- def _record_todo_display(self, result: ToolReturnValue) -> None: @@ -592,12 +600,12 @@ def running_prompt_accepts_submission(self) -> bool: def should_handle_running_prompt_key(self, key: str) -> bool: if key in {"c-o", "c-e"}: return self.has_expandable_panel() + if key == "escape": + return self._cancel_event is not None if self._current_approval_request_panel is not None: return key in {"up", "down", "enter", "1", "2", "3", "4"} if self._turn_ended: return False - if key == "escape": - return self._cancel_event is not None if key == "c-t": return bool(getattr(self, "_latest_todos", ())) # ↑ on empty buffer: recall last queued message. diff --git a/src/pythinker_code/ui/shell/visualize/_live_view.py b/src/pythinker_code/ui/shell/visualize/_live_view.py index 4b7f5fb1..7a45ed9b 100644 --- a/src/pythinker_code/ui/shell/visualize/_live_view.py +++ b/src/pythinker_code/ui/shell/visualize/_live_view.py @@ -28,6 +28,10 @@ from pythinker_code.session_recap import build_turn_recap_line from pythinker_code.soul import format_token_count +from pythinker_code.soul.live_tokens import ( + get_turn_output_tokens, + snapshot_output_tokens_for_turn, +) from pythinker_code.tools.display import DiffDisplayBlock, TodoDisplayBlock, TodoDisplayItem from pythinker_code.ui.shell.components.render_utils import ( cell_width, @@ -644,7 +648,7 @@ def _working_indicator(self) -> RenderableType: ActivitySnapshot( label=spinner_message(now), elapsed_s=elapsed, - tokens=getattr(self, "_latest_context_tokens", None) or 0, + tokens=get_turn_output_tokens(), token_rate=self._turn_token_rate(now), ), width=width, @@ -659,11 +663,11 @@ def _working_indicator(self) -> RenderableType: def _turn_token_rate(self, now: float) -> int | None: """Stable recent tokens/sec for the running turn, or None until known. - Samples the cumulative context token counter at refresh cadence and + Samples the session-wide output-token counter at refresh cadence and derives the rate over a short sliding window, so the readout tracks live throughput instead of a whole-turn average. """ - tokens = getattr(self, "_latest_context_tokens", None) or 0 + tokens = get_turn_output_tokens() # Lazy init: subclasses used in tests don't always run __init__. samples = getattr(self, "_turn_token_samples", None) if samples is None: @@ -687,8 +691,9 @@ def _todo_activity_line( ) -> Text: label = _todo_activity_label(label) parts = [format_elapsed(elapsed_s)] - if self._latest_context_tokens: - parts.append(f"↓ {format_token_count(self._latest_context_tokens)} tokens") + turn_tokens = get_turn_output_tokens() + if turn_tokens: + parts.append(f"↓ {format_token_count(turn_tokens)} tokens") rate = self._turn_token_rate(time.monotonic()) if rate: parts.append(f"{rate} t/s") @@ -850,6 +855,12 @@ def compose(self, *, include_status: bool = True) -> RenderableType: blocks.append(self._status_block.render()) return Group(*blocks) + def _begin_turn_token_window(self) -> None: + """Start a fresh per-turn token-rate window: reset the baseline snapshot + and drop prior-turn samples so the t/s rate can't be skewed by stale data.""" + snapshot_output_tokens_for_turn() + self._turn_token_samples.clear() + def dispatch_wire_message(self, msg: WireMessage) -> None: """Dispatch the Wire message to UI components.""" assert not isinstance(msg, StepInterrupted) # handled in visualize_loop @@ -862,6 +873,7 @@ def dispatch_wire_message(self, msg: WireMessage) -> None: if self._active_turn_depth == 0: self._active_turn_depth = 1 self._turn_start_time = time.monotonic() + self._begin_turn_token_window() self.refresh_soon() return if isinstance(msg, StepRetry): @@ -873,6 +885,7 @@ def dispatch_wire_message(self, msg: WireMessage) -> None: case TurnBegin(user_input=user_input): if self._active_turn_depth == 0: self._turn_start_time = time.monotonic() + self._begin_turn_token_window() self._recap_user_input = ( user_input if isinstance(user_input, str) @@ -901,6 +914,9 @@ def dispatch_wire_message(self, msg: WireMessage) -> None: case CompactionBegin(): self._compaction_block = _CompactionBlock( context_tokens=self._latest_context_tokens, + todos_renderable=self._pinned_todo_block( + width=current_console_width(), hide_active=False, elapsed_s=0.0 + ), ) self.refresh_soon() case CompactionEnd(): diff --git a/src/pythinker_code/utils/io.py b/src/pythinker_code/utils/io.py index 8e916be2..3170ff07 100644 --- a/src/pythinker_code/utils/io.py +++ b/src/pythinker_code/utils/io.py @@ -4,10 +4,41 @@ import json import os import tempfile +from collections.abc import Generator from pathlib import Path from typing import Any +@contextlib.contextmanager +def file_lock(path: Path) -> Generator[None]: + """Cross-process exclusive lock for read-modify-write cycles on *path*. + + ``atomic_json_write`` prevents torn files but not lost updates: two processes + that both load before either saves drop each other's changes. Wrap the whole + load → mutate → save in this lock to serialize concurrent writers. The lock + file (``.lock``) is kept on disk — unlinking would split the lock across + inodes. On platforms without ``fcntl`` (Windows), this is a no-op. Blocking + (flock + small JSON I/O) — call via ``asyncio.to_thread`` from event-loop code. + """ + lock_file = path.with_name(path.name + ".lock") + lock_file.parent.mkdir(parents=True, exist_ok=True) + fh = lock_file.open("a+", encoding="utf-8") + try: + try: + import fcntl + except ImportError: + yield + else: + fcntl.flock(fh.fileno(), fcntl.LOCK_EX) + try: + yield + finally: + with contextlib.suppress(OSError): + fcntl.flock(fh.fileno(), fcntl.LOCK_UN) + finally: + fh.close() + + def ends_with_newline(path: Path) -> bool: """True if *path* is missing/empty or its last byte is a newline. diff --git a/src/pythinker_code/utils/mcp_names.py b/src/pythinker_code/utils/mcp_names.py new file mode 100644 index 00000000..0fe154d3 --- /dev/null +++ b/src/pythinker_code/utils/mcp_names.py @@ -0,0 +1,60 @@ +"""Shared MCP server/tool key normalization (mcpext-6 / task 4.6).""" + +from __future__ import annotations + +import hashlib +import re +from typing import Any, cast + +from pythinker_code.exception import MCPConfigError + +_MCP_NAME_MAX = 64 +_UNSAFE_CHARS = re.compile(r"[^A-Za-z0-9_.-]+") + + +def normalize_mcp_server_name(name: str) -> str: + """Return a deterministic, path-safe MCP server key.""" + trimmed = name.strip() + if not trimmed: + raise MCPConfigError("MCP server name must not be empty") + cleaned = _UNSAFE_CHARS.sub("_", trimmed).strip("._-") + if not cleaned: + digest = hashlib.sha256(trimmed.encode("utf-8")).hexdigest()[:8] + cleaned = f"server_{digest}" + if len(cleaned) <= _MCP_NAME_MAX: + return cleaned + digest = hashlib.sha256(trimmed.encode("utf-8")).hexdigest()[:8] + head = cleaned[: _MCP_NAME_MAX - 9] + return f"{head}_{digest}" + + +def mcp_tool_runtime_key(server_name: str, tool_name: str) -> str: + """Build the runtime registry key for an MCP tool.""" + server = normalize_mcp_server_name(server_name) + safe_tool = _UNSAFE_CHARS.sub("_", tool_name.strip()).strip("._-") or "tool" + return f"mcp__{server}__{safe_tool}" + + +def normalize_mcp_servers_in_config(config: dict[str, Any]) -> dict[str, Any]: + """Re-key ``mcpServers`` entries to normalized names; fail on collisions. + + Mutates ``config`` in place (reassigns ``config["mcpServers"]``) and returns + the same dict for chaining. + """ + servers = config.get("mcpServers") + if not isinstance(servers, dict): + return config + typed_servers = cast(dict[str, Any], servers) + normalized: dict[str, Any] = {} + raw_by_norm: dict[str, str] = {} + for raw_name, server_config in typed_servers.items(): + norm = normalize_mcp_server_name(raw_name) + if norm in normalized: + other = raw_by_norm[norm] + raise MCPConfigError( + f"MCP server name collision: '{raw_name}' and '{other}' both normalize to '{norm}'" + ) + raw_by_norm[norm] = raw_name + normalized[norm] = server_config + config["mcpServers"] = normalized + return config diff --git a/src/pythinker_code/utils/media_limits.py b/src/pythinker_code/utils/media_limits.py new file mode 100644 index 00000000..c5ee479f --- /dev/null +++ b/src/pythinker_code/utils/media_limits.py @@ -0,0 +1,20 @@ +"""Provider-safe media attachment limits (task 7.5).""" + +from __future__ import annotations + +# Conservative defaults aligned with common multimodal provider caps. +MAX_IMAGE_BYTES = 20 * 1024 * 1024 +MAX_VIDEO_BYTES = 100 * 1024 * 1024 +MAX_IMAGE_PIXELS = 20_000_000 + + +def format_byte_limit(limit_bytes: int) -> str: + """Human-readable size for user-facing errors.""" + if limit_bytes <= 0: + # Invalid/disabled limit: surface the raw value, not a misleading "0 KB". + return f"{limit_bytes} bytes" + if limit_bytes >= 1024 * 1024: + return f"{limit_bytes // (1024 * 1024)} MB" + if limit_bytes >= 1024: + return f"{limit_bytes // 1024} KB" + return f"{limit_bytes} bytes" diff --git a/src/pythinker_code/wire/server.py b/src/pythinker_code/wire/server.py index 18d967d6..553716a2 100644 --- a/src/pythinker_code/wire/server.py +++ b/src/pythinker_code/wire/server.py @@ -37,6 +37,7 @@ QuestionResponse, Request, StatusUpdate, + TextPart, ToolCallRequest, is_event, is_request, @@ -741,9 +742,31 @@ async def _handle_prompt( error=JSONRPCErrorObject(code=ErrorCodes.CHAT_PROVIDER_ERROR, message=str(e)), ) except MaxStepsReached as e: + handoff: str | None = None + if isinstance(self._soul, PythinkerSoul): + from pythinker_code.soul.btw import generate_max_steps_handoff + + try: + handoff = await generate_max_steps_handoff(self._soul) + except Exception: + logger.warning("Max-steps handoff failed", exc_info=True) + handoff = None + if handoff: + await self._send_msg( + JSONRPCEventMessage( + method="event", + params=TextPart(text=f"\n── handoff ──\n{handoff}"), + ) + ) + result: dict[str, JsonType] = { + "status": Statuses.MAX_STEPS_REACHED, + "steps": e.n_steps, + } + if handoff: + result["handoff"] = handoff return JSONRPCSuccessResponse( id=msg.id, - result={"status": Statuses.MAX_STEPS_REACHED, "steps": e.n_steps}, + result=result, ) except RunCancelled: return JSONRPCSuccessResponse( diff --git a/tasks/blackbox-port-status.md b/tasks/blackbox-port-status.md new file mode 100644 index 00000000..011ab22f --- /dev/null +++ b/tasks/blackbox-port-status.md @@ -0,0 +1,146 @@ +# Blackbox Reference Port — Live Status Ledger + +> Reactivated 2026-06-15 on `feat/agent-behaviour-tweaks`. Source plan: +> `docs/superpowers/plans/agent_enhancment.md`. Reference tree (read-only): +> `blackbox/pythinker-src/`. + +## Legend + +| Column | Meaning | +| --- | --- | +| `gap_id` | Roadmap gap or task id | +| `phase` | Roadmap phase | +| `reference_path` | Blackbox source (or `missing-reference`) | +| `target_paths` | Pythinker modules | +| `status` | `todo`, `verify-existing`, `in_progress`, `done`, `skipped`, `future-approved-only` | +| `test_gate` | Focused test command or phase gate | +| `notes` | Decision, rationale, de-scope proof | + +## Whole-Source Reverse Engineering Inventory + +| reference_area | reference_paths | target_paths | decision | status | tests_or_skip_gate | +| --- | --- | --- | --- | --- | --- | +| Runtime loop | `query.ts`, `query/` | `soul/pythinkersoul.py`, `soul/context.py`, `soul/compaction.py` | adapt | todo | Phase 2 gate | +| Tool contract | `services/tools/`, `tools/**` | `soul/toolset.py`, `tools/**`, `hooks/**` | adopt/adapt | todo | Phase 2 gate | +| CLI/bootstrap | `entrypoints/cli.tsx`, `main.tsx` | `cli/__init__.py`, `app.py`, `ui/shell/` | adapt; skip Bun/Ink | skip | De-scope: Typer/Rich native | +| Non-interactive transports | `cli/print.ts`, `entrypoints/sdk/` | `ui/print/`, `wire/`, `acp/` | adapt; skip CCR | skip | De-scope: CCR remote transport | +| Terminal UI | `screens/`, `components/`, `ink/` | `ui/shell/`, `wire/` | native-equivalent; skip Ink | verify-existing | Phase 7 gate | +| Slash commands | `commands/**` | `soul/slash.py`, `ui/shell/slash.py` | adapt | todo | Phase 5 | +| Keybindings | `keybindings/`, `components/PromptInput/` | `ui/shell/keymap.py`, `ui/shell/prompt.py` | adapt | todo | Phase 5.11 | +| Permissions UI | `components/permissions/`, `utils/permissions/` | `soul/permission.py`, `soul/approval.py`, `approval_runtime/` | adopt invariants | verify-existing | Phase 1.3–1.4 | +| Background tasks | `Task.ts`, `tasks/**` | `background/`, `tools/background/` | adapt; skip remote/dream | adapt | native-equivalent partial | +| Services core | `services/tools/`, `services/compact/` | `soul/toolset.py`, `soul/compaction.py` | adopt/adapt | todo | Phase 2 | +| MCP | `services/mcp/**` | `soul/toolset.py`, `tools/mcp_resource/`, `cli/mcp.py` | adopt portable | verify-existing | Phase 4; live list_changed refresh | +| Memory | `memdir/**` | `project_memory.py`, `memory/`, `tools/memory/`, `tools/recall/` | adopt hygiene | verify-existing | Phase 3 | +| Skills | `skills/**` | `skill/__init__.py`, `tools/skill/`, `skills/**` | adapt | todo | Phase 3.4–3.8 | +| Agents/subagents | `tools/AgentTool/**` | `agentspec.py`, `subagents/`, `tools/agent/` | adapt | todo | Phase 5 | +| Hooks | `utils/hooks/**` | `hooks/**` | adapt | todo | Phase 5.8 | +| Plugins | `plugins/**` | `plugin/`, `cli/plugin.py` | adapt if approved | todo | Phase 3.9 | +| Settings | `utils/settings/**` | `config.py` | native-equivalent | verify-existing | Phase 7.7 audit | +| Output styles | `outputStyles/**` | dynamic injections | skip | skipped | De-scope: not Pythinker goal | +| Auth/providers | `utils/auth.ts`, `services/oauth/**` | `auth/**`, `llm.py` | native-equivalent | verify-existing | Phase 7.6 audit | +| Constants/schemas | `constants/**`, `schemas/**` | `agents/default/system.md`, `wire/types.py` | adapt | verify-existing | Phase 1.1 | +| Native TS | `native-ts/file-index/` | `ui/shell/prompt.py` | adapt ideas | done | Phase 7.4 | +| File/search UX | `tools/GrepTool`, `tools/FileReadTool` | `tools/file/**` | adapt | verify-existing | Phase 1.2 | +| LSP | `services/lsp/**` | none | future-approved-only | future-approved-only | Too large without approval | +| Sandbox | `utils/sandbox/**` | none | future-approved-only | future-approved-only | Phase 7.8 decision | +| Computer/voice/buddy | `voice/**`, `buddy/**` | none | skip | skipped | Product-only features | +| Remote/bridge | `bridge/**`, `remote/**` | `acp/`, `wire/` | native-equivalent IDE | skipped | CCR not a goal | +| Migrations | `migrations/**` | `config.py` helpers | native-equivalent | verify-existing | Phase 7.7 audit | +| Scratch/runtime | `.pythinker/scratch/` | `scratchpad.py` | native-equivalent | done | Existing | +| Harness/evals | `services/vcr.ts` | `tests_e2e/`, `tests_ai/` | adapt | todo | Phase 6 | + +## Missing / Ambiguous Reference Artifacts + +| artifact | status | substitute | +| --- | --- | --- | +| `blackbox/pythinker-src/src/query/transitions.ts` | missing-reference | `query/stopHooks.ts`, `query/tokenBudget.ts`, `query.ts` | +| `blackbox/pythinker-src/src/skills/mcpSkills.js` | missing-reference | Do not infer; audit `skill/__init__.py` only | + +## Roadmap Task Status (61 tasks) + +| gap_id | phase | reference_path | target_paths | status | test_gate | notes | +| --- | ---: | --- | --- | --- | --- | --- | +| 1.1 | 1 | `constants/prompts.ts`, `utils/messages.ts`, `utils/xml.ts` | `agents/default/system.md`, `utils/trust.py` | done | `tests/utils/test_trust.py`, `tests/core/test_default_agent.py` | Verified 2026-06-15; focused tests pass | +| 1.2 | 1 | `utils/messages.ts`, `tools/` | `tools/web/*`, `tools/file/grep_local.py`, `tools/shell/`, `tools/mcp_resource/`, `tools/recall/` | done | `tests/tools/test_untrusted_wrapping.py` | Verified 2026-06-15; wrapping tests pass | +| 1.3 | 1 | `utils/permissions/` | `tools/file/__init__.py`, `soul/approval.py` | done | `tests/core/test_approval_auto.py` | Verified 2026-06-15; config/dangerous edit gates covered | +| 1.4 | 1 | `utils/permissions/permissions.ts` | `soul/permission.py`, `approval_runtime/` | done | `tests/core/test_approval_auto.py`, `tests/core/test_permission_profiles.py` | Verified 2026-06-15; 102 approval tests pass | +| 2.1 | 2 | `query.ts`, `services/tools/toolExecution.ts` | `tools/utils.py`, `tools/shell/` | done | `tests/utils/test_result_builder.py` | Verified; spill + recovery hints | +| 2.2 | 2 | compaction reference | `soul/compaction.py`, `soul/context.py` | done | `tests/core/test_context_pruning.py` | prune_stale_tool_outputs present | +| 2.3 | 2 | tool descriptions | `tools/**/*.md`, `tools/agent/` | done | `tests/tools/test_tool_descriptions.py` | when-to-use in tool .md files | +| 2.4 | 2 | `query.ts` | `soul/pythinkersoul.py` | verify-existing | soul recovery tests | orphan repair, truncation nudge | +| 2.5 | 2 | hooks/permissions | `hooks/engine.py`, `soul/permission.py` | verify-existing | `tests/hooks/test_engine.py` | PreToolUse block tests | +| 2.6 | 2 | `toolExecution.ts` | `soul/toolset.py` | done | `tests/core/test_toolset.py` | tool input snapshots isolated | +| 2.7 | 2 | `toolOrchestration.ts` | `soul/toolset.py` | done | `tests/core/test_toolset.py` | Shell remains exclusive; no parallel cascade | +| 2.8 | 2 | `context.ts` | `soul/dynamic_injections/git_status.py` | done | `tests/core/test_git_status_injection_provider.py` | Reuses collect_git_context | +| 2.9 | 2 | compaction | `soul/compaction.py` | done | `tests/core/test_simple_compaction.py` | multi-round tool pair regression | +| 2.10 | 2 | `query.ts` | `soul/pythinkersoul.py`, `config.py` | done | `tests/core/test_pythinkersoul_retry_recovery.py` | compaction_failed handoff | +| 2.11 | 2 | compaction cleanup | `soul/pythinkersoul.py`, `soul/context.py` | verify-existing | `tests/core/test_dynamic_injection_hooks.py`, `tests/core/test_compaction_restore.py` | provider rearm + restore invariants | +| 2.12 | 2 | `tools.ts` | `soul/toolset.py` | done | `tests/core/test_toolset.py` | deterministic MCP publish order | +| 3.1 | 3 | memdir recall | `tools/recall/__init__.py` | done | `tests/tools/test_recall.py` | search/read + bounded read windows | +| 3.2 | 3 | `memdir/**` | `config.py`, `project_memory.py`, `memory/` | done | `tests/core/test_project_memory.py`, `tests/core/test_memory_phase_bcd.py` | durable memory remains opt-in | +| 3.3 | 3 | working-set recall | `memory/recall.py` | done | `tests/core/test_memory_phase_bcd.py` | re-arms on working-set shift + memory mtime | +| 3.4 | 3 | `skills/loadSkillsDir.ts` | `skill/__init__.py`, `tools/skill/` | done | `tests/tools/test_skill_tool.py` | resource manifests present | +| 3.5 | 3 | bundled skills | `skills/customize-pythinker/`, `skills/agent-creator/` | done | `tests/core/test_builtin_authoring_skills.py` | config + agent authoring skills present | +| 3.6 | 3 | `memoryScan.ts` | `memory/recall.py`, `project_memory.py` | done | `tests/core/test_memory_phase_bcd.py` | lexical manifest scan with source/mtime | +| 3.7 | 3 | memory prompts | `agents/default/system.md`, `memory/recall.py` | done | `tests/core/test_memory_phase_bcd.py`, prompt audit | memory is background/stale reference | +| 3.8 | 3 | skill frontmatter | `skill/__init__.py`, `tools/skill/` | done | `tests/core/test_skill.py` | adopts name/description/type/scope; ignores non-Pythinker fields | +| 3.9 | 3 | `plugins/**` | `skill/__init__.py`, `plugin/` | done | ledger audit | plugins expose tools/config/skill roots; marketplace/output styles skipped | +| 4.1 | 4 | `services/mcp/client.ts` | `tools/mcp_resource/` | done | `tests/tools/test_mcp_resource.py` | resources + prompts listed; reads untrusted | +| 4.2 | 4 | MCP live refresh | `cli/mcp.py`, `soul/toolset.py`, `ui/shell/slash.py` | done | `tests/core/test_mcp_lifecycle.py` | Persistent MCP sessions plus `tools/list_changed` / resources / prompts handlers call `refresh_mcp_server`; manual `/mcp refresh` remains | +| 4.3 | 4 | MCP docker stdio | `soul/toolset.py`, `cli/mcp.py` | done | `tests/core/test_mcp_docker_rm.py`, `tests/core/test_mcp_cleanup.py`, `tests/tools/test_mcp_startup_timeout.py` | --rm on add + config load via prepare_mcp_config_dict | +| 4.4 | 4 | MCP prompts | `tools/mcp_resource/`, `agents/default/agent.yaml` | done | `tests/tools/test_mcp_resource.py` | InvokeMcpPrompt returns untrusted messages | +| 4.5 | 4 | `elicitationHandler.ts` | `wire/`, `acp/` | future-approved-only | — | Needs Wire contract approval | +| 4.6 | 4 | MCP naming | `utils/mcp_names.py`, `soul/toolset.py`, `cli/mcp.py` | done | `tests/core/test_mcp_name_normalization.py` | Server key normalization + collision errors at load | +| 4.7 | 4 | MCP OAuth | `soul/toolset.py`, `cli/mcp.py` | done | `tests/tools/test_mcp_startup_timeout.py` | OAuth servers skip unauthorized with auth hint; hosted/XAA skipped | +| 5.1 | 5 | plan mode | `soul/permission.py`, subagents | verify-existing | `tests/core/test_permission_profiles.py` | profile downgrade + shell denial; MCP/plugin child E2E thin | +| 5.2 | 5 | subagent usage | `subagents/`, `tools/agent/` | verify-existing | `tests/subagents/test_usage_rollup.py` | batch resume double-count regression test added | +| 5.3 | 5 | plan tool | `tools/plan/`, `soul/dynamic_injections/plan_mode.py` | done | `tests/tools/test_tool_descriptions.py`, `tests/core/test_plan_mode_injection_provider.py` | written plans must include verification | +| 5.4 | 5 | `TodoWriteTool` | `tools/todo/` | done | `tests/tools/test_todo.py` | cancelled status persists and renders distinctly | +| 5.5 | 5 | progress UI | `tools/progress/` | done | `tests/tools/test_progress.py` | ProgressNote producer exists; description anti-spam | +| 5.6 | 5 | suggestions | `tools/suggest/`, `ui/shell/prompt.py` | done | `tests/tools/test_suggest.py` | emit/render + Alt+S accept→prefill | +| 5.7 | 5 | ACP questions | `acp/`, `tools/ask_user/` | done | `tests/acp/test_session_question.py` | unsupported ACP clients get unsupported/fallback semantics | +| 5.8 | 5 | `schemas/hooks.ts` | `hooks/events.py` | verify-existing | `tests/e2e/test_hooks_wire_e2e.py`, `tests/tools/test_agent_tool.py` | supported lifecycle events include PostCompact/SessionEnd/SubagentStart/SubagentStop/Notification; prompt/HTTP hooks skipped | +| 5.9 | 5 | markdown agents | `subagents/discovery.py` | verify-existing | `tests/core/test_subagent_discovery.py` | max_turns/steps + disallowed_tools/exclude_tools mapped | +| 5.10 | 5 | `processUserInput/` | `ui/shell/` | verify-existing | `tests/ui_and_conv/test_shell_slash_commands.py`, `tests/utils/test_slash_command.py` | native slash and shell-mode routing covered | +| 5.11 | 5 | `keybindings/` | `ui/shell/keymap.py` | verify-existing | `tests/ui_and_conv/test_keymap_thinking.py`, `test_tui_card_keymap.py`, `test_slash_completer.py` | Registry + slash fuzzy completion covered; ctrl+r history search deferred | +| 5.12 | 5 | REPL tips | `ui/shell/` | future-approved-only | — | static tips require UX approval; no analytics | +| 5.13 | 5 | suggestions fork | `tools/suggest/` | done | `tests/tools/test_suggest.py` | uses Suggestion tool/event; speculation fork skipped | +| 6.1 | 6 | telemetry tree | `telemetry/` | done | `tests/core/test_otel_span_tree.py`, `tests/telemetry/test_telemetry.py`, `tests/telemetry/test_otel_resource.py` | connected spans, GenAI attribute plumbing, telemetry-off no-op | +| 6.2 | 6 | `services/vcr.ts` | `tests_e2e/` | future-approved-only | — | requires explicit cassette/redaction design | +| 6.3 | 6 | eval harness | `tests_ai/eval_gate.py`, `tests_e2e/eval_schema.py` | verify-existing | `tests/test_eval_harness_wiring.py`, `tests_e2e/test_eval_schema.py` | Offline schema + report budget gate; Harbor live metrics deferred | +| 6.4 | 6 | failure thresholds | `soul/pythinkersoul.py`, `config.py` | done | `tests/core/test_pythinkersoul_stuck_loop.py` | stuck/failure-threshold handoff with reset behavior | +| 6.5 | 6 | max steps | `soul/pythinkersoul.py`, `soul/btw.py`, `wire/server.py`, `acp/session.py` | done | `tests/core/test_max_steps_handoff.py` | shell/print/wire/ACP handoff on MaxStepsReached | +| 6.6 | 6 | telemetry sanitize | `telemetry/names.py`, `soul/toolset.py` | done | `tests/telemetry/test_tool_name_sanitize.py` | Span/metric labels sanitized; runtime tool names unchanged | +| 6.7 | 6 | VCR fixtures | `tests_e2e/` | future-approved-only | — | depends on Task 6.2 cassette design | +| 7.1 | 7 | model defense | `soul/dynamic_injections/model_defense.py` | verify-existing | injection tests | Model-keyed defense | +| 7.2 | 7 | `screens/REPL.tsx` | `ui/shell/` | verify-existing | shell tests | Native-equivalent shell surfaces exist across live view, prompt, slash, keymap, pickers, approvals, questions, suggestions, and task/model/session/MCP panels. Ink/Buddy/cost/idle/message-selector surfaces remain de-scoped | +| 7.3 | 7 | Ink engine | `ui/shell/` | skipped | — | De-scope: no Pi-TUI replacement | +| 7.4 | 7 | `native-ts/file-index/` | `ui/shell/prompt.py` | done | prompt tests | File mention polish: source paths now rank ahead of equally relevant test paths; native-ts indexer, .rgignore expansion, and async refresh remain separate future work | +| 7.5 | 7 | media limits | `utils/media_limits.py`, `tools/file/read_media.py` | done | `tests/utils/test_media_limits.py` | Per-kind byte caps + image pixel ceiling | +| 7.6 | 7 | `utils/auth.ts` | `auth/**` | verify-existing | auth tests | Native-equivalent: managed provider registry, OAuthManager precedence/refresh, and llm.py ambient-key guards. Skipped apiKeyHelper, CCR managed-session isolation, Pythoughts OAuth/beta router | +| 7.7 | 7 | `migrations/**` | `config.py` | verify-existing | config tests | Native-equivalent: idempotent JSON→TOML, github_repo, OAuth keyring→file migrations plus scoped TOML merge/locks. No migration_version registry until a real Pythinker deprecation ships | +| 7.8 | 7 | `utils/sandbox/**` | none | future-approved-only | — | Sandbox decision | +| 7.9 | 7 | session search | `tools/recall/` | done | `tests/tools/test_recall.py` | title/id/plan_slug lexical search and output documented; branch/tag ranking deferred until sessions persist branch/tag metadata | + +## Explicit De-Scope (skip proof) + +| area | decision | rationale | +| --- | --- | --- | +| GrowthBook/Statsig, new telemetry endpoints | skip | Opt-out OTel only; no new hosted telemetry per AGENTS.md | +| pythinkerai hosted MCP, TEAMMEM, KAIROS, voice, buddy | skip | Product-hosted; outside CLI goals | +| React/Ink renderer, Yoga layout | skip | Pythinker uses Rich/prompt_toolkit | +| Output styles directory | skip | Use dynamic injections only if approved | +| CCR remote bridge | skip | ACP/wire cover IDE integration | +| Pi-TUI engine replacement | skip | Phase 7.3 explicit | +| Blackbox skill-only frontmatter (`allowed-tools`, `disable-model-invocation`, hooks/context/path/shell metadata) | skip | Pythinker skill loader intentionally keeps skills as instructional resources; agent/tool execution fields live in agent specs, hooks, and config | +| Plugin marketplace, plugin agents, plugin MCP expansion, plugin output styles | skip | Current Pythinker plugin scope is local tools/config plus skill-root discovery; expansion needs product approval | + +## Phase Exit Gates + +| phase | gate | status | +| ---: | --- | --- | +| 0 | Ledger complete; maintainers see remain/skip | done | +| 1 | `make check-pythinker-code && make test-pythinker-code` | verify-existing done; branch UI ANSI/welcome test fixes landed | +| 2 | focused + check; full test if shared context changed | pending | +| 3–8 | per plan dashboard | pending | diff --git a/tasks/clean-code-guard-followups.md b/tasks/clean-code-guard-followups.md new file mode 100644 index 00000000..695fd9c4 --- /dev/null +++ b/tasks/clean-code-guard-followups.md @@ -0,0 +1,106 @@ +# Clean-code-guard follow-ups — feat/agent-behaviour-tweaks + +Deferred findings from the deep scan of `git diff 1ad0339c127c...HEAD`. +Do not land in the current PR; track here for follow-up. + +## Status + +- [x] Type the new `report`-block parser (`src/pythinker_code/ui/shell/tool_renderers/agent.py:114-124`). + Resolved: replaced ad-hoc `Any` with `TypedDict` (`_ReportFinding`, `_ReportBlock`) + `cast`. + `uv run pyright src/pythinker_code/ui/shell/tool_renderers/agent.py` → 0 errors. +- [x] Format the two unformatted test files + (`tests/ui_and_conv/test_review_findings_parser.py`, + `tests/ui_and_conv/test_md_repair_characterization.py`). + Resolved: `uv run ruff format` → both clean. + +## Deferred — medium severity + +### 1. `disconnect_mcp_server` swallows close errors and overwrites `info.error` (C03 + C04) + +- Location: `src/pythinker_code/soul/toolset.py:1378-1385` +- Mechanism: `except Exception` catches `asyncio.TimeoutError` (subclass of `Exception` since + Python 3.11) from `asyncio.wait_for(info.client.close(), ...)` and logs at DEBUG, then + unconditionally sets `info.status = "failed"` and `info.error = "disconnected"` — clobbering + any pre-existing connect/refresh error. +- Impact: a user disconnecting a hung MCP server sees `error="disconnected"` with no + actionable cause. `mcp_status_snapshot` cannot distinguish a clean user-initiated disconnect + from a close timeout. +- Suggested fix: catch `(TimeoutError, ClientError)` (or the specific client class used here) + explicitly, log at WARNING, and only overwrite `info.error` when no prior error is recorded. +- Acceptance: a new test feeds a hanging `info.client.close()` and asserts + `mcp_status_snapshot(...)["error"]` reflects the timeout, not the literal string + `"disconnected"`. + +### 2. `reconnect_mcp_server` re-raises the raw exception, not the classified error string + +- Location: `src/pythinker_code/soul/toolset.py:1418-1426` +- Mechanism: `_connect_mcp_server` sets `info.error = _classify_mcp_connect_error(e, ...)`, + but the call site raises `MCPRuntimeError(f"Failed to reconnect MCP server '{server_name}': + {error}")` where `error` is the original `Exception` object, not `info.error`. +- Impact: the user sees an unwrapped traceback message instead of the structured classification + that `mcp_status_snapshot` already shows. +- Suggested fix: raise with `MCPRuntimeError(info.error or str(error))`. +- Acceptance: a new test monkey-patches `_connect_mcp_server` to raise, asserts the + re-raised `MCPRuntimeError` message starts with the classified string. + +### 3. `mcp_tool_runtime_key` normalizes server name — silent public contract change + +- Location: `src/pythinker_code/utils/mcp_names.py:31-35` and + `src/pythinker_code/soul/toolset.py:553, 1294, 1365` +- Mechanism: keys are now `mcp____` (e.g. `my server/v2` → + `myserverv2`). Previous contract was `mcp____`. +- Impact: agent yamls or hand-written tool references that hardcoded raw unsafe keys will + silently fail to resolve. The `Runtime.mcp_tools` docstring (`src/pythinker_code/soul/agent.py:237-238`) + still says raw `mcp____`. +- Suggested fix: update the `Runtime.mcp_tools` docstring to note normalization, and add a + regression test that registers a server named e.g. `my server/v2` and asserts + `runtime.mcp_tools` carries the normalized key. +- Acceptance: docstring + test pass; AGENTS.md "Preserve public compatibility" rule satisfied. + +### 4. `read_media` description claims 100 MB max, but images are now capped at 20 MB + +- Location: `src/pythinker_code/tools/file/read_media.md:10` and + `src/pythinker_code/tools/file/read_media.py:69` +- Mechanism: `read_media.md` interpolates `MAX_MEDIA_MEGABYTES = 100` (from + `utils/media_limits.py: max(MAX_IMAGE_BYTES=20MB, MAX_VIDEO_BYTES=100MB) // 1MB`). + But images are capped at 20 MB (plus a 20 MP pixel limit in + `tools/file/read_media.py:69`). +- Impact: a model reading the description will try a 50 MB image and get a runtime + `ToolError(... exceeds the max 20 MB limit for image files)`. +- Suggested fix: document per-kind limits in `read_media.md` + (`{MAX_IMAGE_MEGABYTES} MB for images / {MAX_MEDIA_MEGABYTES} MB for video`) or + expose `MAX_IMAGE_MEGABYTES` separately. +- Acceptance: `make check-tools` (or `uv run pytest tests/tools/test_read_media.py`) passes + and the description matches the runtime error string for the first oversized image read. + +## Deferred — low severity + +- `InvokeMcpPrompt` hides exception class in user-facing error + (`src/pythinker_code/tools/mcp_resource/__init__.py:172-179`). Prepend + `type(exc).__name__` to the message. +- `_publish_connected_mcp_tools` comment claims "configured order" but uses dict insertion + order (`src/pythinker_code/soul/toolset.py:539-549`). Reword comment to + "deterministic per session — dict insertion order". +- `_render_prompt_messages` does not branch on multimodal content + (`src/pythinker_code/tools/mcp_resource/__init__.py:133-142`); text-only is the common case. + +## Verification gates I ran + +- `uv run pyright src/pythinker_code/ui/shell/tool_renderers/agent.py` → 0 errors (was 4). +- `uv run ruff format --check` on the two test files → clean. +- `uv run ruff check` on dirty files → clean. +- `uv run pytest tests/ui_and_conv/test_review_findings_parser.py tests/ui_and_conv/test_markdown_guards.py tests/ui_and_conv/test_md_repair_characterization.py` → 52/52 passed. +- `git diff 1ad0339c127c...HEAD -- pyproject.toml uv.lock` → empty (no new deps; + AGENTS.md zero-new-bundled-deps satisfied). +- `git diff 1ad0339c127c...HEAD -- src/ | grep "^+.*except Exception"` → 5 new bare + `except Exception`; the only load-bearing C03 tripwire is `disconnect_mcp_server` + (item 1 above). + +## Residual risk + +- The `_RE_REPORT_BLOCK` regex (`.*?` lazy) is untested for nested-`report` blocks. + LLM output rarely produces this; low real-world risk. Add a regression test if observed. +- Multi-server `mcp_status_snapshot` race when disconnect + reconnect overlap (concurrent + path; needs a focused stress test). +- Whether any external user config or agent yaml hardcodes raw + `mcp____` keys (depends on user-side state, not in repo). diff --git a/tests/cli/test_plugin_marketplace_cli.py b/tests/cli/test_plugin_marketplace_cli.py new file mode 100644 index 00000000..aa110725 --- /dev/null +++ b/tests/cli/test_plugin_marketplace_cli.py @@ -0,0 +1,102 @@ +"""CLI tests for the `pythinker plugin marketplace` command group.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest +from typer.testing import CliRunner + +from pythinker_code.cli.plugin import cli + +runner = CliRunner() + + +@pytest.fixture(autouse=True) +def _share_dir(tmp_path: Path, monkeypatch): + monkeypatch.setenv("PYTHINKER_SHARE_DIR", str(tmp_path / "share")) + + +def _make_directory_marketplace(root: Path) -> Path: + manifest = root / ".claude-plugin" / "marketplace.json" + manifest.parent.mkdir(parents=True) + manifest.write_text( + json.dumps( + {"name": "mk", "plugins": [{"name": "demo", "version": "1.0.0", "source": "./demo"}]} + ), + encoding="utf-8", + ) + pm = root / "demo" / ".claude-plugin" / "plugin.json" + pm.parent.mkdir(parents=True) + pm.write_text(json.dumps({"name": "demo", "version": "1.0.0"}), encoding="utf-8") + return root + + +def test_marketplace_add_list_remove(tmp_path: Path) -> None: + market = _make_directory_marketplace(tmp_path / "market") + + result = runner.invoke(cli, ["marketplace", "add", str(market), "--name", "mk"]) + assert result.exit_code == 0, result.output + assert "Added marketplace 'mk'" in result.output + + listed = runner.invoke(cli, ["marketplace", "list"]) + assert "mk" in listed.output + + removed = runner.invoke(cli, ["marketplace", "remove", "mk"]) + assert removed.exit_code == 0 + assert "mk" not in runner.invoke(cli, ["marketplace", "list"]).output + + +def test_marketplace_install_and_installed_and_uninstall(tmp_path: Path) -> None: + market = _make_directory_marketplace(tmp_path / "market") + runner.invoke(cli, ["marketplace", "add", str(market), "--name", "mk"]) + + installed = runner.invoke(cli, ["marketplace", "install", "demo", "mk"]) + assert installed.exit_code == 0, installed.output + assert "Installed 'demo@mk'" in installed.output + + listing = runner.invoke(cli, ["marketplace", "installed"]) + assert "demo@mk" in listing.output + + # name@marketplace form also works for uninstall. + removed = runner.invoke(cli, ["marketplace", "uninstall", "demo@mk"]) + assert removed.exit_code == 0, removed.output + assert "demo@mk" not in runner.invoke(cli, ["marketplace", "installed"]).output + + +def test_marketplace_install_unknown_errors(tmp_path: Path) -> None: + result = runner.invoke(cli, ["marketplace", "install", "demo", "ghost"]) + assert result.exit_code == 1 + assert "not configured" in result.output + + +def test_marketplace_refresh(tmp_path: Path) -> None: + market = _make_directory_marketplace(tmp_path / "market") + runner.invoke(cli, ["marketplace", "add", str(market), "--name", "mk"]) + result = runner.invoke(cli, ["marketplace", "refresh", "mk"]) + assert result.exit_code == 0, result.output + assert "1 plugin(s) available" in result.output + + +def test_marketplace_add_derives_name_from_github(tmp_path: Path) -> None: + result = runner.invoke(cli, ["marketplace", "add", "anthropics/claude-plugins-official"]) + assert result.exit_code == 0, result.output + assert "claude-plugins-official" in runner.invoke(cli, ["marketplace", "list"]).output + + +def test_plugin_disable_then_enable_roundtrip(tmp_path: Path) -> None: + from pythinker_code.config import load_config + + result = runner.invoke(cli, ["disable", "ponytail"]) + assert result.exit_code == 0, result.output + assert "Disabled plugin 'ponytail'" in result.output + assert "ponytail" in load_config().plugins.disabled + + # Idempotent: disabling again is a no-op. + assert "already disabled" in runner.invoke(cli, ["disable", "ponytail"]).output + + result = runner.invoke(cli, ["enable", "ponytail"]) + assert result.exit_code == 0, result.output + assert "Enabled plugin 'ponytail'" in result.output + assert "ponytail" not in load_config().plugins.disabled diff --git a/tests/conftest.py b/tests/conftest.py index 80d8467c..715e3f53 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -48,6 +48,7 @@ from pythinker_code.llm import ALL_MODEL_CAPABILITIES, LLM from pythinker_code.metadata import WorkDirMeta from pythinker_code.notifications import NotificationManager +from pythinker_code.plugin import loader as plugin_loader from pythinker_code.session import Session from pythinker_code.session_state import SessionState from pythinker_code.soul.agent import BuiltinSystemPromptArgs, LaborMarket, Runtime @@ -136,6 +137,22 @@ def _isolate_share_dir( monkeypatch.setenv("PYTHINKER_SHARE_DIR", str(tmp_path_factory.mktemp("share"))) +@pytest.fixture(autouse=True) +def _isolate_external_plugins(monkeypatch: pytest.MonkeyPatch) -> None: + """Neutralize Claude/Codex plugin discovery so the suite never reads the + developer's real ``~/.claude``/``~/.codex`` plugins. + + External plugin auto-detection (``plugins.discover_external`` defaults on) + makes agent/skill/command discovery scan those roots. Left unpatched, + discovery results would depend on whatever plugins a given machine has + installed, making tests non-deterministic across dev boxes and CI. Tests + that exercise external discovery monkeypatch these roots to their own temp + dirs, which overrides this default within their own scope. + """ + monkeypatch.setattr(plugin_loader, "claude_plugin_roots", lambda: []) + monkeypatch.setattr(plugin_loader, "codex_plugin_roots", lambda: []) + + @pytest.fixture def builtin_args(temp_work_dir: HostPath) -> BuiltinSystemPromptArgs: """Create builtin arguments with temporary work directory.""" diff --git a/tests/core/test_agent_spec.py b/tests/core/test_agent_spec.py index 98b6365b..3f8a9834 100644 --- a/tests/core/test_agent_spec.py +++ b/tests/core/test_agent_spec.py @@ -62,6 +62,7 @@ def test_load_default_agent_spec(): "pythinker_code.tools.web:FetchURL", "pythinker_code.tools.mcp_resource:ListMcpResources", "pythinker_code.tools.mcp_resource:ReadMcpResource", + "pythinker_code.tools.mcp_resource:InvokeMcpPrompt", "pythinker_code.tools.plan:ExitPlanMode", "pythinker_code.tools.plan.enter:EnterPlanMode", ] @@ -271,6 +272,7 @@ def test_load_default_agent_spec(): "pythinker_code.tools.web:FetchURL", "pythinker_code.tools.mcp_resource:ListMcpResources", "pythinker_code.tools.mcp_resource:ReadMcpResource", + "pythinker_code.tools.mcp_resource:InvokeMcpPrompt", "pythinker_code.tools.plan:ExitPlanMode", "pythinker_code.tools.plan.enter:EnterPlanMode", ] @@ -402,6 +404,7 @@ def test_load_default_agent_spec(): "pythinker_code.tools.web:FetchURL", "pythinker_code.tools.mcp_resource:ListMcpResources", "pythinker_code.tools.mcp_resource:ReadMcpResource", + "pythinker_code.tools.mcp_resource:InvokeMcpPrompt", "pythinker_code.tools.plan:ExitPlanMode", "pythinker_code.tools.plan.enter:EnterPlanMode", ] @@ -543,6 +546,7 @@ def test_load_default_agent_spec(): "pythinker_code.tools.web:FetchURL", "pythinker_code.tools.mcp_resource:ListMcpResources", "pythinker_code.tools.mcp_resource:ReadMcpResource", + "pythinker_code.tools.mcp_resource:InvokeMcpPrompt", "pythinker_code.tools.plan:ExitPlanMode", "pythinker_code.tools.plan.enter:EnterPlanMode", ] @@ -669,6 +673,7 @@ def test_load_default_agent_spec(): "pythinker_code.tools.web:FetchURL", "pythinker_code.tools.mcp_resource:ListMcpResources", "pythinker_code.tools.mcp_resource:ReadMcpResource", + "pythinker_code.tools.mcp_resource:InvokeMcpPrompt", "pythinker_code.tools.plan:ExitPlanMode", "pythinker_code.tools.plan.enter:EnterPlanMode", ] @@ -843,6 +848,7 @@ def test_load_agent_spec_default_extension(): "pythinker_code.tools.web:FetchURL", "pythinker_code.tools.mcp_resource:ListMcpResources", "pythinker_code.tools.mcp_resource:ReadMcpResource", + "pythinker_code.tools.mcp_resource:InvokeMcpPrompt", "pythinker_code.tools.plan:ExitPlanMode", "pythinker_code.tools.plan.enter:EnterPlanMode", ] diff --git a/tests/core/test_config.py b/tests/core/test_config.py index 3f50f443..71f55e3e 100644 --- a/tests/core/test_config.py +++ b/tests/core/test_config.py @@ -51,6 +51,7 @@ def test_default_config_dump(): "max_steps_per_turn": 1000, "max_consecutive_failures": 8, "max_truncation_recoveries": 3, + "max_compaction_failures": 1, "max_session_cost_usd": None, "budget_nudge_ratio": 0.75, "max_retries_per_step": 3, @@ -105,10 +106,18 @@ def test_default_config_dump(): }, "hooks": [], "merge_all_available_skills": True, + "plugins": { + "discover_external": True, + "external_exec": False, + "enabled": [], + "disabled": [], + "options": {}, + }, "extra_skill_dirs": [], "telemetry": True, "session_retention_days": 30, "skip_auto_prompt_injection": False, + "git_status_injection": True, "tui": { "style": "card", "prompt_history_enabled": True, @@ -242,6 +251,16 @@ def test_load_config_invalid_ralph_iterations(): load_config_from_string('{"loop_control": {"max_ralph_iterations": -2}}') +def test_load_config_max_compaction_failures_too_low(): + with pytest.raises(ConfigError, match="max_compaction_failures"): + load_config_from_string('{"loop_control": {"max_compaction_failures": 0}}') + + +def test_load_config_git_status_injection_false(): + config = load_config_from_string('{"git_status_injection": false}') + assert config.git_status_injection is False + + def test_load_config_reserved_context_size(): config = load_config_from_string('{"loop_control": {"reserved_context_size": 30000}}') assert config.loop_control.reserved_context_size == 30000 diff --git a/tests/core/test_default_agent.py b/tests/core/test_default_agent.py index c519e0d9..6470f11c 100644 --- a/tests/core/test_default_agent.py +++ b/tests/core/test_default_agent.py @@ -335,6 +335,7 @@ async def test_default_agent_background_bash_guardrails(runtime: Runtime): "FetchURL", "ListMcpResources", "ReadMcpResource", + "InvokeMcpPrompt", "EnterPlanMode", ] ) diff --git a/tests/core/test_exit_plan_mode_verification.py b/tests/core/test_exit_plan_mode_verification.py new file mode 100644 index 00000000..c22dfd41 --- /dev/null +++ b/tests/core/test_exit_plan_mode_verification.py @@ -0,0 +1,12 @@ +from pythinker_code.tools.plan.__init__ import _plan_lacks_verification_section + + +def test_plan_lacks_verification_section_detects_missing_heading() -> None: + assert _plan_lacks_verification_section("## Plan\nDo the thing.\n") + assert not _plan_lacks_verification_section("## Plan\n## Verification\nmake test\n") + + +# The ExitPlanMode description assertion lives in +# tests/tools/test_tool_descriptions.py::test_exit_plan_mode_description_requires_verification_section +# (it checks both the "Verification section" heading and the "smallest command, +# test, or check" guidance). This file is scoped to the _plan_lacks_verification_section helper. diff --git a/tests/core/test_git_status_injection_provider.py b/tests/core/test_git_status_injection_provider.py new file mode 100644 index 00000000..1233a430 --- /dev/null +++ b/tests/core/test_git_status_injection_provider.py @@ -0,0 +1,94 @@ +"""Git status dynamic injection for the root agent.""" + +from __future__ import annotations + +from unittest.mock import AsyncMock, MagicMock, patch + +from pythinker_code.soul.dynamic_injections.git_status import GitStatusInjectionProvider + + +def _make_soul(*, is_subagent: bool = False, git_status_injection: bool = True) -> MagicMock: + soul = MagicMock() + soul.is_subagent = is_subagent + soul.runtime.work_dir = "/tmp/repo" + soul.runtime.config.git_status_injection = git_status_injection + return soul + + +class TestGitStatusInjectionProvider: + async def test_injects_bounded_snapshot_on_first_step(self) -> None: + provider = GitStatusInjectionProvider() + soul = _make_soul() + git_block = ( + "\n" + "Working directory: /tmp/repo\n" + "Branch: main\n" + "Dirty files (1):\n" + " M src/foo.py\n" + "" + ) + with patch( + "pythinker_code.soul.dynamic_injections.git_status.collect_git_context", + new_callable=AsyncMock, + return_value=git_block, + ): + result = await provider.get_injections([], soul) + + assert len(result) == 1 + assert result[0].type == "git_status" + assert "may be stale" in result[0].content + assert "Branch: main" in result[0].content + assert "M src/foo.py" in result[0].content + assert "" not in result[0].content + + async def test_skips_when_not_a_git_repo(self) -> None: + provider = GitStatusInjectionProvider() + with patch( + "pythinker_code.soul.dynamic_injections.git_status.collect_git_context", + new_callable=AsyncMock, + return_value="", + ): + assert await provider.get_injections([], _make_soul()) == [] + + async def test_does_not_reinject_unchanged_snapshot(self) -> None: + provider = GitStatusInjectionProvider() + soul = _make_soul() + git_block = "\nBranch: main\n" + with patch( + "pythinker_code.soul.dynamic_injections.git_status.collect_git_context", + new_callable=AsyncMock, + return_value=git_block, + ): + assert len(await provider.get_injections([], soul)) == 1 + assert await provider.get_injections([], soul) == [] + + async def test_reinjects_after_compaction(self) -> None: + provider = GitStatusInjectionProvider() + soul = _make_soul() + git_block = "\nBranch: main\n" + with patch( + "pythinker_code.soul.dynamic_injections.git_status.collect_git_context", + new_callable=AsyncMock, + return_value=git_block, + ): + await provider.get_injections([], soul) + await provider.on_context_compacted() + assert len(await provider.get_injections([], soul)) == 1 + + async def test_subagents_are_excluded(self) -> None: + provider = GitStatusInjectionProvider() + with patch( + "pythinker_code.soul.dynamic_injections.git_status.collect_git_context", + new_callable=AsyncMock, + ) as mock_collect: + assert await provider.get_injections([], _make_soul(is_subagent=True)) == [] + mock_collect.assert_not_called() + + async def test_respects_config_disable(self) -> None: + provider = GitStatusInjectionProvider() + with patch( + "pythinker_code.soul.dynamic_injections.git_status.collect_git_context", + new_callable=AsyncMock, + ) as mock_collect: + assert await provider.get_injections([], _make_soul(git_status_injection=False)) == [] + mock_collect.assert_not_called() diff --git a/tests/core/test_live_tokens.py b/tests/core/test_live_tokens.py new file mode 100644 index 00000000..436f39de --- /dev/null +++ b/tests/core/test_live_tokens.py @@ -0,0 +1,52 @@ +"""Unit tests for the session-wide live output-token accumulator.""" + +from __future__ import annotations + +from collections.abc import Iterator + +import pytest + +from pythinker_code.soul.live_tokens import ( + add_total_output_tokens, + get_total_output_tokens, + get_turn_output_tokens, + reset_for_tests, + snapshot_output_tokens_for_turn, +) + + +@pytest.fixture(autouse=True) +def _isolate() -> Iterator[None]: + reset_for_tests() + yield + reset_for_tests() + + +def test_total_accumulates_across_sources() -> None: + # Every in-process soul (main, subagent, background) funnels here. + add_total_output_tokens(50) # main agent step + add_total_output_tokens(30) # subagent step + add_total_output_tokens(20) # background step + assert get_total_output_tokens() == 100 + + +def test_turn_delta_excludes_pre_turn_tokens() -> None: + add_total_output_tokens(100) # produced before the turn began + snapshot_output_tokens_for_turn() + assert get_turn_output_tokens() == 0 + add_total_output_tokens(40) # main + subagent work during the turn + assert get_turn_output_tokens() == 40 + assert get_total_output_tokens() == 140 + + +def test_turn_delta_never_negative_without_snapshot() -> None: + # No snapshot taken: baseline is 0, delta tracks the total. + add_total_output_tokens(25) + assert get_turn_output_tokens() == 25 + + +def test_non_positive_counts_are_ignored() -> None: + add_total_output_tokens(10) + add_total_output_tokens(0) + add_total_output_tokens(-5) + assert get_total_output_tokens() == 10 diff --git a/tests/core/test_max_steps_handoff.py b/tests/core/test_max_steps_handoff.py index f23a2df4..95a75839 100644 --- a/tests/core/test_max_steps_handoff.py +++ b/tests/core/test_max_steps_handoff.py @@ -11,8 +11,10 @@ import asyncio from dataclasses import dataclass, field -from unittest.mock import MagicMock, patch +from pathlib import Path +from unittest.mock import AsyncMock, MagicMock, patch +import pytest from pythinker_core.message import Message, ToolCall from pythinker_core.tooling import ToolError, ToolResult @@ -100,3 +102,61 @@ async def fake_step(provider, sys_prompt, toolset, history, **kw): summary = asyncio.run(generate_max_steps_handoff(soul)) assert summary is None + + +@pytest.mark.asyncio +async def test_wire_server_streams_handoff_on_max_steps( + runtime, tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """Wire clients get a handoff event and result field when the step ceiling hits.""" + from pythinker_core.tooling.simple import SimpleToolset + + from pythinker_code.soul import MaxStepsReached + from pythinker_code.soul.agent import Agent + from pythinker_code.soul.context import Context + from pythinker_code.soul.pythinkersoul import PythinkerSoul + from pythinker_code.wire.jsonrpc import JSONRPCPromptMessage, JSONRPCSuccessResponse + from pythinker_code.wire.server import WireServer + + agent = Agent( + name="Max Steps Wire Test", + system_prompt="sys", + toolset=SimpleToolset(), + runtime=runtime, + ) + soul = PythinkerSoul(agent, context=Context(file_backend=tmp_path / "history.jsonl")) + server = WireServer(soul) + + async def fake_run_soul(*_args, **_kwargs) -> None: + raise MaxStepsReached(3) + + monkeypatch.setattr("pythinker_code.wire.server.run_soul", fake_run_soul) + + sent_events: list[object] = [] + original_send = server._send_msg + + async def capture_send(msg): # type: ignore[no-untyped-def] + if getattr(msg, "method", None) == "event": + sent_events.append(msg.params) + return await original_send(msg) + + monkeypatch.setattr(server, "_send_msg", capture_send) + + with patch( + "pythinker_code.soul.btw.generate_max_steps_handoff", + new=AsyncMock(return_value="Resume with step Z."), + ): + response = await server._handle_prompt( + JSONRPCPromptMessage( + id="1", + params=JSONRPCPromptMessage.Params(user_input="hello"), + ) + ) + + assert isinstance(response, JSONRPCSuccessResponse) + assert response.result == { + "status": "max_steps_reached", + "steps": 3, + "handoff": "Resume with step Z.", + } + assert any(getattr(event, "text", "").endswith("Resume with step Z.") for event in sent_events) diff --git a/tests/core/test_mcp_cli_names.py b/tests/core/test_mcp_cli_names.py new file mode 100644 index 00000000..32713572 --- /dev/null +++ b/tests/core/test_mcp_cli_names.py @@ -0,0 +1,48 @@ +"""MCP CLI name resolution round-trip (task 4.6 / mcpext-6).""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest +from typer.testing import CliRunner + +from pythinker_code.cli.mcp import _resolve_mcp_server_key, cli + + +@pytest.fixture +def mcp_config_file(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path: + monkeypatch.setenv("PYTHINKER_SHARE_DIR", str(tmp_path)) + config_path = tmp_path / "mcp.json" + config_path.write_text( + json.dumps( + { + "mcpServers": { + "my_server_v2": {"command": "echo", "args": ["hello"]}, + } + } + ), + encoding="utf-8", + ) + return config_path + + +def test_remove_accepts_display_name_with_spaces_and_slashes( + mcp_config_file: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + runner = CliRunner() + result = runner.invoke(cli, ["remove", "my server/v2"]) + assert result.exit_code == 0, result.output + saved = json.loads(mcp_config_file.read_text(encoding="utf-8")) + assert saved["mcpServers"] == {} + + +def test_resolve_mcp_server_key_maps_raw_name_to_stored_key() -> None: + servers = {"my_server_v2": {"command": "echo"}} + assert _resolve_mcp_server_key("my server/v2", servers) == "my_server_v2" + + +def test_resolve_mcp_server_key_accepts_already_normalized_name() -> None: + servers = {"context7": {"url": "https://example.com/mcp", "transport": "http"}} + assert _resolve_mcp_server_key("context7", servers) == "context7" diff --git a/tests/core/test_mcp_docker_rm.py b/tests/core/test_mcp_docker_rm.py index cf17f1dc..098000d4 100644 --- a/tests/core/test_mcp_docker_rm.py +++ b/tests/core/test_mcp_docker_rm.py @@ -14,7 +14,7 @@ import pytest -from pythinker_code.cli.mcp import ensure_docker_rm +from pythinker_code.cli.mcp import apply_docker_rm_to_mcp_config_dict, ensure_docker_rm @pytest.mark.parametrize("cmd", ["docker", "podman"]) @@ -42,6 +42,20 @@ def test_handles_empty_args() -> None: assert ensure_docker_rm("docker", []) == [] +def test_apply_docker_rm_to_mcp_config_dict() -> None: + config = apply_docker_rm_to_mcp_config_dict( + { + "mcpServers": { + "ctx": { + "command": "docker", + "args": ["run", "-i", "ghcr.io/example/mcp"], + } + } + } + ) + assert config["mcpServers"]["ctx"]["args"] == ["run", "--rm", "-i", "ghcr.io/example/mcp"] + + def test_full_path_runtime_is_recognized() -> None: # A docker binary referenced by path should still be treated as docker. args = ensure_docker_rm("/usr/bin/docker", ["run", "img"]) diff --git a/tests/core/test_mcp_lifecycle.py b/tests/core/test_mcp_lifecycle.py new file mode 100644 index 00000000..f692aadd --- /dev/null +++ b/tests/core/test_mcp_lifecycle.py @@ -0,0 +1,309 @@ +"""mcpext-2: per-server MCP disconnect/reconnect/refresh.""" + +from __future__ import annotations + +import asyncio +from types import SimpleNamespace +from typing import Any, cast +from unittest.mock import AsyncMock + +import pytest + +from pythinker_code.exception import MCPRuntimeError +from pythinker_code.soul.toolset import ( + MCPServerInfo, + MCPTool, + PythinkerToolset, + _make_mcp_live_refresh_handler, +) + + +def _runtime() -> Any: + from pathlib import Path + + return SimpleNamespace( + mcp_tools={}, + config=SimpleNamespace( + mcp=SimpleNamespace( + client=SimpleNamespace(startup_timeout_ms=1000, tool_call_timeout_ms=1000) + ) + ), + session=SimpleNamespace(dir=Path("/tmp")), + ) + + +def _fake_mcp_tool(server: str, name: str) -> MCPTool[Any]: + mcp_tool = cast(Any, SimpleNamespace(name=name, description="", inputSchema={})) + client = cast(Any, SimpleNamespace()) + return MCPTool(server, mcp_tool, client, runtime=_runtime()) + + +@pytest.mark.asyncio +async def test_disconnect_unregisters_tools_and_marks_disconnected() -> None: + toolset = PythinkerToolset() + runtime = _runtime() + tool = _fake_mcp_tool("alpha", "ToolA") + toolset._mcp_servers["alpha"] = MCPServerInfo( + status="connected", + client=cast(Any, SimpleNamespace(close=AsyncMock())), + tools=[tool], + resources=[], + prompts=[], + server_config={"command": "echo"}, + ) + toolset.add(tool) + runtime.mcp_tools["mcp__alpha__ToolA"] = tool + + await toolset.disconnect_mcp_server("alpha", runtime) + + assert toolset.find("ToolA") is None + assert "mcp__alpha__ToolA" not in runtime.mcp_tools + assert toolset._mcp_servers["alpha"].status == "failed" + assert toolset._mcp_servers["alpha"].error == "disconnected" + + +@pytest.mark.asyncio +async def test_refresh_relists_tools_for_connected_server(monkeypatch: pytest.MonkeyPatch) -> None: + toolset = PythinkerToolset() + runtime = _runtime() + old_tool = _fake_mcp_tool("alpha", "OldTool") + new_tool = _fake_mcp_tool("alpha", "NewTool") + info = MCPServerInfo( + status="connected", + client=cast(Any, SimpleNamespace()), + tools=[old_tool], + resources=[], + prompts=[], + ) + toolset._mcp_servers["alpha"] = info + toolset.add(old_tool) + runtime.mcp_tools["mcp__alpha__OldTool"] = old_tool + + # _inventory_mcp_server discovers without mutating; the caller assigns the + # returned inventory only after the awaited call succeeds. + async def _inventory(_server: str, _server_info: MCPServerInfo, _runtime: Any) -> Any: + return [new_tool], [], [] + + monkeypatch.setattr(toolset, "_inventory_mcp_server", _inventory) + + await toolset.refresh_mcp_server("alpha", runtime) + + assert info.tools == [new_tool] + assert toolset.find("OldTool") is None + assert toolset.find("NewTool") is new_tool + assert runtime.mcp_tools["mcp__alpha__NewTool"] is new_tool + + +@pytest.mark.asyncio +async def test_reconnect_requires_stored_config() -> None: + toolset = PythinkerToolset() + runtime = _runtime() + toolset._mcp_servers["alpha"] = MCPServerInfo( + status="failed", + client=cast(Any, SimpleNamespace(close=AsyncMock())), + tools=[], + resources=[], + prompts=[], + server_config=None, + ) + + with pytest.raises(MCPRuntimeError, match="stored config"): + await toolset.reconnect_mcp_server("alpha", runtime) + + +@pytest.mark.asyncio +async def test_tool_list_changed_triggers_refresh(monkeypatch: pytest.MonkeyPatch) -> None: + import mcp.types + + toolset = PythinkerToolset() + runtime = _runtime() + toolset._mcp_servers["alpha"] = MCPServerInfo( + status="connected", + client=cast(Any, SimpleNamespace()), + tools=[], + resources=[], + prompts=[], + ) + refreshed: list[str] = [] + + async def _refresh(server_name: str, _runtime: Any) -> None: + refreshed.append(server_name) + + monkeypatch.setattr(toolset, "refresh_mcp_server", _refresh) + + class _FakeClient: + pass + + handler = _make_mcp_live_refresh_handler(_FakeClient(), toolset, runtime, "alpha") + + await handler.on_tool_list_changed(mcp.types.ToolListChangedNotification()) + + assert refreshed == ["alpha"] + + +@pytest.mark.asyncio +async def test_disconnect_close_timeout_surfaces_actionable_error( + monkeypatch: pytest.MonkeyPatch, +) -> None: + toolset = PythinkerToolset() + runtime = _runtime() + info = MCPServerInfo( + status="connected", + client=cast(Any, SimpleNamespace(close=AsyncMock())), + tools=[], + resources=[], + prompts=[], + server_config={"command": "echo"}, + ) + toolset._mcp_servers["alpha"] = info + monkeypatch.setattr("pythinker_code.soul.toolset._MCP_CLOSE_TIMEOUT_S", 0.01) + + async def _slow_close() -> None: + await asyncio.sleep(1) + + info.client.close = AsyncMock(side_effect=_slow_close) + + await toolset.disconnect_mcp_server("alpha", runtime) + + assert info.status == "failed" + assert info.error is not None + assert "timed out" in info.error + + +@pytest.mark.asyncio +async def test_reconnect_raises_classified_error(monkeypatch: pytest.MonkeyPatch) -> None: + toolset = PythinkerToolset() + runtime = _runtime() + info = MCPServerInfo( + status="failed", + client=cast(Any, SimpleNamespace(close=AsyncMock())), + tools=[], + resources=[], + prompts=[], + server_config={"command": "missing-binary"}, + ) + toolset._mcp_servers["alpha"] = info + + async def _connect( + _server_name: str, server_info: MCPServerInfo, _runtime: Any + ) -> tuple[str, Exception | None]: + server_info.status = "failed" + server_info.error = "command not found: missing-binary — check the server command/path" + return "alpha", FileNotFoundError("missing-binary") + + monkeypatch.setattr(toolset, "_connect_mcp_server", _connect) + monkeypatch.setattr( + "pythinker_code.soul.toolset._configure_mcp_client_handlers", + lambda *args, **kwargs: None, + ) + + class _FakeClient: + pass + + monkeypatch.setattr("fastmcp.Client", lambda *args, **kwargs: _FakeClient()) + + with pytest.raises(MCPRuntimeError, match="command not found"): + await toolset.reconnect_mcp_server("alpha", runtime) + + +@pytest.mark.asyncio +async def test_refresh_failure_preserves_live_tools(monkeypatch: pytest.MonkeyPatch) -> None: + """A failed refresh must not drop the server's still-working tools.""" + toolset = PythinkerToolset() + runtime = _runtime() + live_tool = _fake_mcp_tool("alpha", "LiveTool") + info = MCPServerInfo( + status="connected", + client=cast(Any, SimpleNamespace()), + tools=[live_tool], + resources=[], + prompts=[], + ) + toolset._mcp_servers["alpha"] = info + toolset.add(live_tool) + runtime.mcp_tools["mcp__alpha__LiveTool"] = live_tool + + async def _failing_inventory(_server: str, _info: MCPServerInfo, _runtime: Any) -> None: + raise RuntimeError("list_tools blew up") + + monkeypatch.setattr(toolset, "_inventory_mcp_server", _failing_inventory) + + with pytest.raises(MCPRuntimeError, match="Failed to refresh"): + await toolset.refresh_mcp_server("alpha", runtime) + + assert toolset.find("LiveTool") is live_tool + assert runtime.mcp_tools["mcp__alpha__LiveTool"] is live_tool + assert info.tools == [live_tool] + + +@pytest.mark.asyncio +async def test_disconnect_reclaims_shadowed_tool_from_other_server() -> None: + """Disconnecting the winning server must fall the tool name back, not orphan it.""" + toolset = PythinkerToolset() + runtime = _runtime() + shared_alpha = _fake_mcp_tool("alpha", "Shared") + shared_beta = _fake_mcp_tool("beta", "Shared") + toolset._mcp_servers["alpha"] = MCPServerInfo( + status="connected", + client=cast(Any, SimpleNamespace(close=AsyncMock())), + tools=[shared_alpha], + resources=[], + prompts=[], + server_config={"command": "echo"}, + ) + toolset._mcp_servers["beta"] = MCPServerInfo( + status="connected", + client=cast(Any, SimpleNamespace(close=AsyncMock())), + tools=[shared_beta], + resources=[], + prompts=[], + server_config={"command": "echo"}, + ) + # Configured order publishes beta last, so it wins the shared name. + toolset._publish_connected_mcp_tools(runtime) + assert toolset.find("Shared") is shared_beta + + await toolset.disconnect_mcp_server("beta", runtime) + + assert toolset.find("Shared") is shared_alpha + assert runtime.mcp_tools["mcp__alpha__Shared"] is shared_alpha + assert "mcp__beta__Shared" not in runtime.mcp_tools + assert toolset._mcp_servers["beta"].status == "failed" + + +@pytest.mark.asyncio +async def test_partial_connect_publishes_connected_servers(monkeypatch: pytest.MonkeyPatch) -> None: + """A server that connects must be published even when a sibling fails the aggregate.""" + from fastmcp.mcp_config import MCPConfig + + toolset = PythinkerToolset() + runtime = _runtime() + good_tool = _fake_mcp_tool("good", "GoodTool") + + async def _connect( + server_name: str, server_info: MCPServerInfo, _runtime: Any + ) -> tuple[str, Exception | None]: + if server_name == "good": + server_info.status = "connected" + server_info.tools = [good_tool] + return server_name, None + server_info.status = "failed" + server_info.error = "boom" + return server_name, RuntimeError("boom") + + monkeypatch.setattr(toolset, "_connect_mcp_server", _connect) + monkeypatch.setattr( + "pythinker_code.soul.toolset._configure_mcp_client_handlers", + lambda *args, **kwargs: None, + ) + monkeypatch.setattr("fastmcp.Client", lambda *args, **kwargs: SimpleNamespace()) + + config = MCPConfig.model_validate( + {"mcpServers": {"good": {"command": "echo"}, "bad": {"command": "echo"}}} + ) + + with pytest.raises(MCPRuntimeError, match="Failed to connect"): + await toolset.load_mcp_tools([config], runtime, in_background=False) + + assert toolset.find("GoodTool") is good_tool + assert runtime.mcp_tools["mcp__good__GoodTool"] is good_tool diff --git a/tests/core/test_mcp_name_normalization.py b/tests/core/test_mcp_name_normalization.py new file mode 100644 index 00000000..246de321 --- /dev/null +++ b/tests/core/test_mcp_name_normalization.py @@ -0,0 +1,52 @@ +"""Tests for MCP server name normalization (task 4.6).""" + +from __future__ import annotations + +import re + +import pytest + +from pythinker_code.exception import MCPConfigError +from pythinker_code.utils.mcp_names import ( + mcp_tool_runtime_key, + normalize_mcp_server_name, + normalize_mcp_servers_in_config, +) + + +def test_normalize_replaces_spaces_and_slashes() -> None: + assert normalize_mcp_server_name("my server/v2") == "my_server_v2" + + +def test_normalize_is_idempotent_for_safe_names() -> None: + assert normalize_mcp_server_name("context7") == "context7" + + +def test_normalize_bounds_overlong_names() -> None: + long_name = "a" * 100 + normalized = normalize_mcp_server_name(long_name) + assert len(normalized) <= 64 + # Truncated names keep a deterministic 8-char hex hash suffix so distinct + # overlong names don't collide (real contract, not a tautology). + assert re.search(r"_[0-9a-f]{8}$", normalized) + assert normalized != normalize_mcp_server_name("b" * 100) + + +def test_empty_name_raises() -> None: + with pytest.raises(MCPConfigError, match="empty"): + normalize_mcp_server_name(" ") + + +def test_collision_surfaces_in_config() -> None: + config = { + "mcpServers": { + "foo bar": {"command": "echo"}, + "foo_bar": {"command": "echo"}, + } + } + with pytest.raises(MCPConfigError, match="collision"): + normalize_mcp_servers_in_config(config) + + +def test_mcp_tool_runtime_key_uses_normalized_server() -> None: + assert mcp_tool_runtime_key("my server", "read_file") == "mcp__my_server__read_file" diff --git a/tests/core/test_plugin_manager.py b/tests/core/test_plugin_manager.py index 24f4a291..f8f1d1b8 100644 --- a/tests/core/test_plugin_manager.py +++ b/tests/core/test_plugin_manager.py @@ -200,32 +200,31 @@ def test_remove_rejects_plugin_root_name(tmp_path: Path): @pytest.mark.asyncio async def test_skill_discovery_includes_plugins_dir(tmp_path: Path, monkeypatch): - """Plugins dir should be included in skill discovery roots.""" + """An installed plugin's ``skills/`` subdir is included in skill roots. + + Plugins live under ``/plugins/cache////`` + and contribute the skills nested inside them — not the plugins root itself. + """ from pythinker_host.path import HostPath from pythinker_code.skill import resolve_skills_roots - plugins_dir = tmp_path / "plugins" - plugins_dir.mkdir() - - # Create a valid plugin with SKILL.md - plugin_dir = plugins_dir / "my-plugin" - plugin_dir.mkdir() - (plugin_dir / "SKILL.md").write_text( - "---\nname: my-plugin\ndescription: test\n---\n# Test", - encoding="utf-8", - ) - (plugin_dir / "plugin.json").write_text( - json.dumps({"name": "my-plugin", "version": "1.0.0"}), - encoding="utf-8", - ) + # Install a plugin in the pythinker cache with a nested skill. + plugin_root = tmp_path / "plugins" / "cache" / "market" / "my-plugin" / "1.0.0" + manifest = plugin_root / ".pythinker-plugin" / "plugin.json" + manifest.parent.mkdir(parents=True) + manifest.write_text(json.dumps({"name": "my-plugin", "version": "1.0.0"}), encoding="utf-8") + skills_dir = plugin_root / "skills" + skill_md = skills_dir / "my-skill" / "SKILL.md" + skill_md.parent.mkdir(parents=True) + skill_md.write_text("---\nname: my-skill\ndescription: test\n---\n# Test", encoding="utf-8") # Point PYTHINKER_SHARE_DIR to tmp_path so get_plugins_dir() returns tmp_path/plugins monkeypatch.setenv("PYTHINKER_SHARE_DIR", str(tmp_path)) scoped = await resolve_skills_roots(HostPath(str(tmp_path))) root_strs = [str(s.root) for s in scoped] - assert str(plugins_dir) in root_strs + assert str(skills_dir.resolve()) in root_strs # --- collect_host_values tests --- diff --git a/tests/core/test_pythinkersoul_retry_recovery.py b/tests/core/test_pythinkersoul_retry_recovery.py index 5e46d38d..c506e3f0 100644 --- a/tests/core/test_pythinkersoul_retry_recovery.py +++ b/tests/core/test_pythinkersoul_retry_recovery.py @@ -560,6 +560,130 @@ async def fake_compact_context() -> None: ] +@pytest.mark.asyncio +async def test_proactive_compaction_failure_yields_handoff( + runtime: Runtime, tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + runtime.config.loop_control.max_compaction_failures = 1 + provider = RecoveringSequenceProvider() + llm = LLM( + chat_provider=provider, + max_context_size=100, + capabilities=set(), + ) + soul, context = _make_soul(runtime, llm, tmp_path) + await context.update_token_count(100_000) + + async def fake_compact_context() -> None: + raise RuntimeError("compact failed") + + monkeypatch.setattr(soul, "compact_context", fake_compact_context) + seen: list[object] = [] + + await run_soul( + soul, + "trigger proactive compaction", + lambda wire: _collect_ui_messages(wire, seen), + asyncio.Event(), + ) + + assert provider.generate_attempts == 0 + final_text = context.history[-1].extract_text(" ") + assert "compaction failed" in final_text.lower() + assert any("compaction failed" in tp.text.lower() for tp in seen if isinstance(tp, TextPart)) + + +@pytest.mark.asyncio +async def test_proactive_compaction_failure_below_threshold_continues_turn( + runtime: Runtime, tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """With max_compaction_failures > 1, a single failed compaction must not abort the turn.""" + runtime.config.loop_control.max_compaction_failures = 2 + provider = RecoveringSequenceProvider() + llm = LLM( + chat_provider=provider, + max_context_size=100, + capabilities=set(), + ) + soul, context = _make_soul(runtime, llm, tmp_path) + await context.update_token_count(100_000) + + async def fake_compact_context() -> None: + raise RuntimeError("compact failed") + + step_ran = False + + async def fake_step() -> object: + nonlocal step_ran + from pythinker_code.soul.pythinkersoul import StepOutcome + + step_ran = True + return StepOutcome( + stop_reason="no_tool_calls", + assistant_message=Message(role="assistant", content="continued after compact miss"), + ) + + monkeypatch.setattr(soul, "compact_context", fake_compact_context) + monkeypatch.setattr(soul, "_step", fake_step) + + await run_soul( + soul, + "trigger proactive compaction", + lambda wire: _collect_ui_messages(wire, []), + asyncio.Event(), + ) + + assert step_ran + assert not any( + "compaction failed" in message.extract_text(" ").lower() for message in context.history + ) + + +@pytest.mark.asyncio +async def test_proactive_compaction_failure_threshold_requires_multiple_failures( + runtime: Runtime, tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + runtime.config.loop_control.max_compaction_failures = 2 + llm = LLM( + chat_provider=RecoveringSequenceProvider(), + max_context_size=100, + capabilities=set(), + ) + soul, context = _make_soul(runtime, llm, tmp_path) + await context.update_token_count(100_000) + step_calls = 0 + + async def fake_compact_context() -> None: + raise RuntimeError("compact failed") + + async def fake_step() -> object | None: + nonlocal step_calls + from pythinker_code.soul.pythinkersoul import StepOutcome + + step_calls += 1 + if step_calls < 2: + return None + return StepOutcome( + stop_reason="no_tool_calls", + assistant_message=Message(role="assistant", content="should not reach"), + ) + + monkeypatch.setattr(soul, "compact_context", fake_compact_context) + monkeypatch.setattr(soul, "_step", fake_step) + + await run_soul( + soul, + "trigger proactive compaction", + lambda wire: _collect_ui_messages(wire, []), + asyncio.Event(), + ) + + assert step_calls == 1 + final_text = context.history[-1].extract_text(" ") + assert "compaction failed" in final_text.lower() + assert "should not reach" not in final_text + + @pytest.mark.asyncio async def test_context_overflow_recovery_is_one_shot_per_turn( runtime: Runtime, tmp_path: Path, monkeypatch: pytest.MonkeyPatch diff --git a/tests/core/test_simple_compaction.py b/tests/core/test_simple_compaction.py index dec99d62..1382d06a 100644 --- a/tests/core/test_simple_compaction.py +++ b/tests/core/test_simple_compaction.py @@ -2,7 +2,7 @@ from inline_snapshot import snapshot from pythinker_core.chat_provider import TokenUsage -from pythinker_core.message import AudioURLPart, ImageURLPart, Message, VideoURLPart +from pythinker_core.message import AudioURLPart, ImageURLPart, Message, ToolCall, VideoURLPart import pythinker_code.prompts as prompts from pythinker_code.soul.compaction import CompactionResult, SimpleCompaction, should_auto_compact @@ -77,6 +77,34 @@ def test_prepare_builds_compact_message_and_preserves_tail(): ) +def test_prepare_preserves_tool_call_rounds_without_orphaning_results(): + first_call = ToolCall( + id="call_1", + function=ToolCall.FunctionBody(name="ReadFile", arguments='{"path":"a.py"}'), + ) + second_call = ToolCall( + id="call_2", + function=ToolCall.FunctionBody(name="ReadFile", arguments='{"path":"b.py"}'), + ) + messages = [ + Message(role="user", content=[TextPart(text="Inspect files")]), + Message(role="assistant", content=[TextPart(text="Reading A")], tool_calls=[first_call]), + Message(role="tool", content=[TextPart(text="A")], tool_call_id="call_1"), + Message(role="assistant", content=[TextPart(text="Reading B")], tool_calls=[second_call]), + Message(role="tool", content=[TextPart(text="B")], tool_call_id="call_2"), + Message(role="assistant", content=[TextPart(text="Done")]), + ] + + result = SimpleCompaction(max_preserved_messages=2).prepare(messages) + + assert [message.role for message in result.to_preserve] == ["assistant", "tool", "assistant"] + assert result.to_preserve[0].tool_calls == [second_call] + assert result.to_preserve[1].tool_call_id == "call_2" + assert [message.role for message in result.to_compact] == ["user", "assistant", "tool"] + assert result.to_compact[1].tool_calls == [first_call] + assert result.to_compact[2].tool_call_id == "call_1" + + # --- CompactionResult.estimated_token_count tests --- diff --git a/tests/core/test_subagent_discovery.py b/tests/core/test_subagent_discovery.py index 6fd4a732..bbfb3960 100644 --- a/tests/core/test_subagent_discovery.py +++ b/tests/core/test_subagent_discovery.py @@ -4,6 +4,7 @@ from unittest.mock import patch import pytest +import yaml from pythinker_host.path import HostPath from pythinker_code.agentspec import DEFAULT_AGENT_FILE @@ -22,6 +23,31 @@ def _write_agent(path: Path, text: str) -> None: path.write_text(text, encoding="utf-8") +def test_parse_markdown_agent_maps_max_turns_and_disallowed_tools(tmp_path: Path) -> None: + path = tmp_path / "worker.md" + spec = parse_markdown_agent( + """--- +name: worker +description: Scoped worker +max_turns: 12 +disallowed_tools: ["Write", "Bash"] +--- +Body +""", + prompt_file=HostPath.unsafe_from_local_path(path), + scope="project", + ) + assert spec.steps == 12 + assert spec.exclude_tools == ( + "pythinker_code.tools.file:WriteFile", + "pythinker_code.tools.shell:Shell", + ) + [type_def] = materialize_markdown_agent_specs([spec], output_dir=tmp_path / "out2") + payload = yaml.safe_load(type_def.agent_file.read_text(encoding="utf-8")) + assert payload["agent"]["steps"] == 12 + assert payload["agent"]["exclude_tools"] == list(spec.exclude_tools or ()) + + @pytest.mark.asyncio async def test_parse_markdown_agent_maps_claude_tools(tmp_path: Path) -> None: path = tmp_path / "planner.md" diff --git a/tests/core/test_toolset.py b/tests/core/test_toolset.py index 9fa48a50..7e2213e8 100644 --- a/tests/core/test_toolset.py +++ b/tests/core/test_toolset.py @@ -11,10 +11,15 @@ import mcp from pydantic import BaseModel -from pythinker_core.tooling import CallableTool2, ToolOk, ToolReturnValue +from pythinker_core.tooling import CallableTool, CallableTool2, ToolOk, ToolReturnValue from pythinker_core.tooling.error import ToolNotFoundError as PythinkerCoreToolNotFoundError -from pythinker_code.soul.toolset import MCPTool, PythinkerToolset, _configure_mcp_client_stderr_log +from pythinker_code.soul.toolset import ( + MCPServerInfo, + MCPTool, + PythinkerToolset, + _configure_mcp_client_stderr_log, +) from pythinker_code.wire.types import ToolCall, ToolResult, ToolUseSkipped @@ -49,6 +54,20 @@ async def __call__(self, params: DummyParams) -> ToolReturnValue: return ToolOk(output="b") +class MutatesNestedInputTool(CallableTool): + name: str = "MutatesNestedInput" + description: str = "Mutates nested input" + parameters: dict[str, Any] = { + "type": "object", + "properties": {"payload": {"type": "object"}}, + "required": ["payload"], + } + + async def __call__(self, payload: dict[str, str]) -> ToolReturnValue: + payload["value"] = "mutated" + return ToolOk(output="mutated") + + def _make_toolset() -> PythinkerToolset: ts = PythinkerToolset() ts.add(DummyToolA()) @@ -60,6 +79,20 @@ def _tool_names(ts: PythinkerToolset) -> set[str]: return {t.name for t in ts.tools} +def _fake_mcp_tool(server: str, name: str) -> MCPTool[Any]: + mcp_tool = cast(Any, SimpleNamespace(name=name, description="", inputSchema={})) + client = cast(Any, SimpleNamespace()) + runtime = cast( + Any, + SimpleNamespace( + config=SimpleNamespace( + mcp=SimpleNamespace(client=SimpleNamespace(tool_call_timeout_ms=1000)) + ) + ), + ) + return MCPTool(server, mcp_tool, client, runtime=runtime) + + # --- hide() --- @@ -81,6 +114,37 @@ def test_hide_returns_false_for_nonexistent_tool(): assert ts.hide("NoSuchTool") is False +def test_mcp_duplicate_tool_publish_order_follows_server_order(): + ts = PythinkerToolset() + runtime = cast(Any, SimpleNamespace(mcp_tools={})) + alpha_tool = _fake_mcp_tool("alpha", "SharedTool") + beta_tool = _fake_mcp_tool("beta", "SharedTool") + ts._mcp_servers = { # pyright: ignore[reportPrivateUsage] + "alpha": MCPServerInfo( + status="connected", + client=cast(Any, SimpleNamespace()), + tools=[alpha_tool], + resources=[], + prompts=[], + ), + "beta": MCPServerInfo( + status="connected", + client=cast(Any, SimpleNamespace()), + tools=[beta_tool], + resources=[], + prompts=[], + ), + } + + ts._publish_connected_mcp_tools(runtime) # pyright: ignore[reportPrivateUsage] + + published = ts.find("SharedTool") + assert isinstance(published, MCPTool) + assert published.mcp_server_name == "beta" + assert runtime.mcp_tools["mcp__alpha__SharedTool"] is alpha_tool + assert runtime.mcp_tools["mcp__beta__SharedTool"] is beta_tool + + def test_hide_is_idempotent(): ts = _make_toolset() ts.hide("ToolA") @@ -215,6 +279,41 @@ async def wait_forever() -> None: assert task.cancelled() +async def test_tool_execution_mutation_does_not_rewrite_hook_input() -> None: + ts = PythinkerToolset() + ts.add(MutatesNestedInputTool()) + post_inputs: list[dict[str, Any]] = [] + + async def trigger(*args: Any, **kwargs: Any) -> list[Any]: + return [] + + def fire_and_forget_trigger(*args: Any, **kwargs: Any) -> None: + if args[0] == "PostToolUse": + post_inputs.append(kwargs["input_data"]["tool_input"]) + + ts._hook_engine = cast( # pyright: ignore[reportPrivateUsage] + Any, + SimpleNamespace( + trigger=trigger, + fire_and_forget_trigger=fire_and_forget_trigger, + ), + ) + + result = ts.handle( + ToolCall( + id="mutating-call", + function=ToolCall.FunctionBody( + name="MutatesNestedInput", + arguments='{"payload":{"value":"original"}}', + ), + ) + ) + assert isinstance(result, asyncio.Task) + _ = await result # drive the task to completion; the awaited value is unused + + assert post_inputs == [{"payload": {"value": "original"}}] + + # --- hide/unhide cycle --- diff --git a/tests/subagents/test_usage_rollup.py b/tests/subagents/test_usage_rollup.py index 25d7a779..659aa34e 100644 --- a/tests/subagents/test_usage_rollup.py +++ b/tests/subagents/test_usage_rollup.py @@ -74,6 +74,14 @@ def test_summarize_batch_sums_children() -> None: assert lines[1] == "total_child_cost_usd: 0.0500" +def test_summarize_batch_counts_each_child_once_not_per_resume_line() -> None: + """Resume metadata in the child body must not inflate token roll-up totals.""" + resumed = _result({EXTRA_INPUT_TOKENS: 80, EXTRA_OUTPUT_TOKENS: 20}) + fresh = _result({EXTRA_INPUT_TOKENS: 20, EXTRA_OUTPUT_TOKENS: 10}) + lines = summarize_batch([resumed, fresh]) + assert lines[0] == "total_child_tokens: 100 in / 30 out" + + def test_summarize_batch_empty_when_no_usage() -> None: results = [_result(None), _result({"unrelated": 1})] assert summarize_batch(results) == [] diff --git a/tests/telemetry/test_tool_name_sanitize.py b/tests/telemetry/test_tool_name_sanitize.py new file mode 100644 index 00000000..b060ecc8 --- /dev/null +++ b/tests/telemetry/test_tool_name_sanitize.py @@ -0,0 +1,26 @@ +"""obs-eval-6: MCP/plugin tool names are sanitized before telemetry export.""" + +from __future__ import annotations + +from pythinker_code.telemetry.names import sanitize_telemetry_tool_name + + +def test_mcp_tool_name_is_bounded_and_safe() -> None: + raw = "mcp__my server!__tool/with/slashes" + sanitized = sanitize_telemetry_tool_name(raw) + assert sanitized.startswith("mcp__") + assert "/" not in sanitized + assert " " not in sanitized + assert len(sanitized) <= 64 + + +def test_builtin_tool_name_passes_through_when_safe() -> None: + assert sanitize_telemetry_tool_name("ReadFile") == "ReadFile" + + +def test_unsafe_builtin_name_is_normalized() -> None: + assert sanitize_telemetry_tool_name("weird tool!") == "weird_tool" + + +def test_empty_name_becomes_unknown() -> None: + assert sanitize_telemetry_tool_name(" ") == "unknown_tool" diff --git a/tests/test_eval_harness_wiring.py b/tests/test_eval_harness_wiring.py new file mode 100644 index 00000000..3880b075 --- /dev/null +++ b/tests/test_eval_harness_wiring.py @@ -0,0 +1,54 @@ +"""Offline wiring for scenario efficiency evals (task 6.3).""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from tests_ai.eval_gate import gate_report, load_eval_cases +from tests_e2e.eval_schema import ObservedMetrics, score_eval_case + + +def test_load_example_eval_cases() -> None: + cases = load_eval_cases(Path("tests_ai/eval_cases.example.json")) + assert cases[0].name == "encoding smoke" + assert cases[0].budget.max_tool_calls == 20 + + +def test_gate_report_scores_optional_metrics() -> None: + cases = load_eval_cases(Path("tests_ai/eval_cases.example.json")) + report = [ + { + "file": "tests_ai/test_encoding_error_handling.md", + "cases": [ + { + "name": "encoding smoke", + "pass": True, + "metrics": { + "tool_calls": 2, + "input_tokens": 100, + "output_tokens": 50, + "tools_used": ["ReadFile"], + }, + } + ], + } + ] + verdicts = gate_report(report, cases) + assert len(verdicts) == 1 + assert verdicts[0].passed + + +def test_gate_report_raises_on_unknown_case_name() -> None: + """A report case with no matching eval case is a contract drift, not a silent skip.""" + cases = load_eval_cases(Path("tests_ai/eval_cases.example.json")) + report = [{"file": "x.md", "cases": [{"name": "does-not-exist", "pass": True}]}] + with pytest.raises(ValueError, match="Unknown eval case name"): + gate_report(report, cases) + + +def test_score_eval_case_flags_budget_breach() -> None: + cases = load_eval_cases(Path("tests_ai/eval_cases.example.json")) + observed = ObservedMetrics(tool_calls=999, input_tokens=10**6) + assert not score_eval_case(cases[0], observed).passed diff --git a/tests/test_plugin_dependency.py b/tests/test_plugin_dependency.py new file mode 100644 index 00000000..5e66e257 --- /dev/null +++ b/tests/test_plugin_dependency.py @@ -0,0 +1,111 @@ +"""Tests for plugin dependency parsing, demotion, and install closure.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from pythinker_code.plugin import install, loader +from pythinker_code.plugin.dependency import parse_plugin_identifier, verify_and_demote +from pythinker_code.plugin.manifest import PluginManifest +from pythinker_code.plugin.marketplace import ( + MarketplaceError, + add_marketplace, + parse_marketplace_input, +) + + +def test_parse_plugin_identifier_first_at_only() -> None: + assert parse_plugin_identifier("foo") == ("foo", None) + assert parse_plugin_identifier("foo@mkt") == ("foo", "mkt") + # Only the first '@' separates; the rest stays with the marketplace part. + assert parse_plugin_identifier("foo@mkt@x") == ("foo", "mkt@x") + + +def test_manifest_strips_version_suffix_and_object_form() -> None: + m = PluginManifest.model_validate( + { + "name": "p", + "dependencies": [ + "bare", + "name@mkt", + "name@mkt@^1.2", # version suffix stripped + {"name": "obj", "marketplace": "mk2"}, + {"name": "obj2"}, + ], + } + ) + assert m.dependencies == ["bare", "name@mkt", "name@mkt", "obj@mk2", "obj2"] + + +def test_verify_and_demote_disables_unsatisfied() -> None: + # b depends on a missing plugin -> demoted; a has no deps -> stays. + demoted, issues = verify_and_demote([("a", []), ("b", ["ghost"])], {"a", "b"}) + assert demoted == {"b"} + assert [(i.plugin, i.dependency, i.reason) for i in issues] == [("b", "ghost", "not-found")] + + +def test_verify_and_demote_cascades_and_marks_not_enabled() -> None: + # c needs b, b needs a, but a is installed-yet-disabled. b demotes (a not + # enabled), then c demotes (b no longer enabled). Marketplace qualifier on the + # dep is matched by name. + plugins = [("a", []), ("b", ["a@mk"]), ("c", ["b"])] + demoted, issues = verify_and_demote(plugins, {"b", "c"}) # a known but disabled + assert demoted == {"b", "c"} + reasons = {i.plugin: i.reason for i in issues} + assert reasons == {"b": "not-enabled", "c": "not-enabled"} + + +def _entry(root: Path, name: str, deps: list[str] | None = None) -> None: + """Create a marketplace 'demo'-style plugin source dir with optional deps.""" + src = root / name + manifest = src / ".claude-plugin" / "plugin.json" + manifest.parent.mkdir(parents=True) + body: dict[str, object] = {"name": name, "version": "1.0.0"} + if deps: + body["dependencies"] = deps + manifest.write_text(json.dumps(body), encoding="utf-8") + + +def _marketplace(root: Path, names: list[str]) -> Path: + mk = root / ".claude-plugin" / "marketplace.json" + mk.parent.mkdir(parents=True) + mk.write_text( + json.dumps( + { + "name": "mk", + "plugins": [{"name": n, "version": "1.0.0", "source": f"./{n}"} for n in names], + } + ), + encoding="utf-8", + ) + return root + + +def test_install_pulls_transitive_dependencies(tmp_path: Path) -> None: + market = _marketplace(tmp_path / "market", ["app", "lib"]) + _entry(market, "app", deps=["lib"]) + _entry(market, "lib") + add_marketplace("mk", parse_marketplace_input(str(market))) + + install.install_plugin_from_marketplace("app", "mk") + + found = {p.name for p in loader.discover_plugins().plugins} + assert {"app", "lib"} <= found # dependency installed alongside the root + + +def test_install_blocks_cross_marketplace_dependency(tmp_path: Path) -> None: + market = _marketplace(tmp_path / "market", ["app"]) + _entry(market, "app", deps=["lib@other"]) + add_marketplace("mk", parse_marketplace_input(str(market))) + + with pytest.raises(MarketplaceError, match="Cross-marketplace"): + install.install_plugin_from_marketplace("app", "mk") + + # The failed install must roll back: "app" was materialized before its + # cross-marketplace dependency was rejected, so it must not be left recorded. + from pythinker_code.plugin.installed import load_installed_plugins + + assert "app@mk" not in load_installed_plugins() diff --git a/tests/test_plugin_install.py b/tests/test_plugin_install.py new file mode 100644 index 00000000..b905ec78 --- /dev/null +++ b/tests/test_plugin_install.py @@ -0,0 +1,187 @@ +"""Tests for installing plugins from marketplaces into the versioned cache.""" + +from __future__ import annotations + +import json +import subprocess +from pathlib import Path + +import pytest + +from pythinker_code.plugin import installed, loader, marketplace +from pythinker_code.plugin.install import install_plugin_from_marketplace +from pythinker_code.plugin.marketplace import MarketplaceError + + +@pytest.fixture(autouse=True) +def _share_dir(tmp_path: Path, monkeypatch): + monkeypatch.setenv("PYTHINKER_SHARE_DIR", str(tmp_path / "share")) + + +def _make_directory_marketplace(root: Path, plugin: str) -> Path: + """A directory marketplace with one plugin at ./plugins/.""" + manifest = root / ".claude-plugin" / "marketplace.json" + manifest.parent.mkdir(parents=True) + manifest.write_text( + json.dumps( + { + "name": "mk", + "plugins": [{"name": plugin, "version": "1.0.0", "source": f"./plugins/{plugin}"}], + } + ), + encoding="utf-8", + ) + plugin_dir = root / "plugins" / plugin + pm = plugin_dir / ".claude-plugin" / "plugin.json" + pm.parent.mkdir(parents=True) + pm.write_text(json.dumps({"name": plugin, "version": "1.0.0"}), encoding="utf-8") + (plugin_dir / "skills" / "s").mkdir(parents=True) + (plugin_dir / "skills" / "s" / "SKILL.md").write_text( + "---\nname: s\ndescription: d\n---\n# s", encoding="utf-8" + ) + return root + + +def test_install_from_directory_marketplace(tmp_path: Path) -> None: + market = _make_directory_marketplace(tmp_path / "market", "demo") + marketplace.add_marketplace("mk", marketplace.parse_marketplace_input(str(market))) + + record = install_plugin_from_marketplace("demo", "mk") + + dest = Path(record.install_path) + assert dest.exists() + assert (dest / ".claude-plugin" / "plugin.json").is_file() + assert (dest / "skills" / "s" / "SKILL.md").is_file() + # Recorded under name@marketplace. + assert "demo@mk" in installed.load_installed_plugins() + + +def test_install_then_discoverable(tmp_path: Path) -> None: + market = _make_directory_marketplace(tmp_path / "market", "demo") + marketplace.add_marketplace("mk", marketplace.parse_marketplace_input(str(market))) + install_plugin_from_marketplace("demo", "mk") + + # The installed plugin now lives in the pythinker cache and is discoverable. + result = loader.discover_plugins(include_external=False) + assert any(p.name == "demo" and p.origin == "pythinker" for p in result.plugins) + + +def test_install_unknown_marketplace_raises(tmp_path: Path) -> None: + with pytest.raises(MarketplaceError, match="not configured"): + install_plugin_from_marketplace("demo", "ghost") + + +def test_install_missing_plugin_raises(tmp_path: Path) -> None: + market = _make_directory_marketplace(tmp_path / "market", "demo") + marketplace.add_marketplace("mk", marketplace.parse_marketplace_input(str(market))) + with pytest.raises(MarketplaceError, match="not found in marketplace"): + install_plugin_from_marketplace("absent", "mk") + + +def test_install_rejects_unsafe_name(tmp_path: Path) -> None: + with pytest.raises(MarketplaceError, match="Unsafe"): + install_plugin_from_marketplace("../evil", "mk") + + +def test_install_traversal_source_blocked(tmp_path: Path) -> None: + root = tmp_path / "market" + manifest = root / ".claude-plugin" / "marketplace.json" + manifest.parent.mkdir(parents=True) + manifest.write_text( + json.dumps({"name": "mk", "plugins": [{"name": "evil", "source": "../../etc"}]}), + encoding="utf-8", + ) + marketplace.add_marketplace("mk", marketplace.parse_marketplace_input(str(root))) + with pytest.raises(MarketplaceError, match="escapes marketplace root"): + install_plugin_from_marketplace("evil", "mk") + + +def test_install_reuses_external_via_symlink(tmp_path: Path, monkeypatch) -> None: + # Plugin already present in an external (Claude) cache -> symlink, no copy. + external_version = tmp_path / "claude_cache" / "mk" / "demo" / "9.9.9" + pm = external_version / ".claude-plugin" / "plugin.json" + pm.parent.mkdir(parents=True) + pm.write_text(json.dumps({"name": "demo", "version": "9.9.9"}), encoding="utf-8") + + from pythinker_code.plugin import install as install_mod + + monkeypatch.setattr( + install_mod, "external_installed_plugin_dirs", lambda m, p: [external_version] + ) + # Marketplace must be configured, but fetch must be skipped by the reuse path. + marketplace.add_marketplace("mk", marketplace.parse_marketplace_input("owner/repo")) + + record = install_plugin_from_marketplace("demo", "mk") + dest = Path(record.install_path) + assert dest.is_symlink() + assert dest.resolve() == external_version.resolve() + assert record.version == "9.9.9" + # Discoverable through the symlink. + result = loader.discover_plugins(include_external=False) + assert any(p.name == "demo" for p in result.plugins) + + +@pytest.mark.parametrize( + "bad_url", + ["ext::sh -c touch${IFS}/tmp/x", "fd::7", "--upload-pack=evil", "/local/path"], +) +def test_git_clone_rejects_unsafe_url(tmp_path: Path, bad_url: str) -> None: + from pythinker_code.plugin.install import _git_clone + + with pytest.raises(MarketplaceError, match="Unsafe or unsupported git URL"): + _git_clone(bad_url, None, tmp_path / "dest") + + +def test_git_clone_rejects_flag_ref(tmp_path: Path) -> None: + from pythinker_code.plugin.install import _git_clone + + with pytest.raises(MarketplaceError, match="Unsafe git ref"): + _git_clone("https://example.com/x.git", "--upload-pack=evil", tmp_path / "dest") + + +def _git_available() -> bool: + try: + subprocess.run(["git", "--version"], capture_output=True, check=True, timeout=10) + return True + except (OSError, subprocess.SubprocessError): + return False + + +@pytest.mark.skipif(not _git_available(), reason="git not available") +def test_install_git_plugin_source_offline(tmp_path: Path) -> None: + # A bare-ish local git repo serving as a plugin source (clone works offline). + repo = tmp_path / "plugrepo" + pm = repo / ".claude-plugin" / "plugin.json" + pm.parent.mkdir(parents=True) + pm.write_text(json.dumps({"name": "gitp", "version": "2.0.0"}), encoding="utf-8") + for cmd in ( + ["git", "init", "-q"], + ["git", "add", "-A"], + ["git", "-c", "user.email=t@t", "-c", "user.name=t", "commit", "-qm", "init"], + ): + subprocess.run(cmd, cwd=repo, check=True, capture_output=True) + + market = tmp_path / "market" + manifest = market / ".claude-plugin" / "marketplace.json" + manifest.parent.mkdir(parents=True) + manifest.write_text( + json.dumps( + { + "name": "mk", + "plugins": [ + { + "name": "gitp", + "version": "2.0.0", + "source": {"source": "git", "url": f"file://{repo}"}, + } + ], + } + ), + encoding="utf-8", + ) + marketplace.add_marketplace("mk", marketplace.parse_marketplace_input(str(market))) + + record = install_plugin_from_marketplace("gitp", "mk") + dest = Path(record.install_path) + assert (dest / ".claude-plugin" / "plugin.json").is_file() + assert not (dest / ".git").exists() # .git excluded diff --git a/tests/test_plugin_integration.py b/tests/test_plugin_integration.py new file mode 100644 index 00000000..d5ab64c7 --- /dev/null +++ b/tests/test_plugin_integration.py @@ -0,0 +1,373 @@ +"""End-to-end: installed plugins contribute artifacts to discovery paths.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest +from pythinker_host.path import HostPath + +from pythinker_code.plugin import integration, loader +from pythinker_code.plugin.policy import PluginPolicy + + +def _install_plugin_with_skill(cache: Path, plugin: str, skill: str) -> Path: + """Create a Claude-style plugin with a nested skill under *cache*.""" + root = cache / plugin / plugin / "1.0.0" + manifest = root / ".claude-plugin" / "plugin.json" + manifest.parent.mkdir(parents=True, exist_ok=True) + manifest.write_text(json.dumps({"name": plugin, "version": "1.0.0"}), encoding="utf-8") + skill_md = root / "skills" / skill / "SKILL.md" + skill_md.parent.mkdir(parents=True, exist_ok=True) + skill_md.write_text( + f"---\nname: {skill}\ndescription: {skill} does things\n---\n# {skill}\n", + encoding="utf-8", + ) + return root + + +@pytest.fixture +def _no_external(monkeypatch): + monkeypatch.setattr(loader, "claude_plugin_roots", lambda: []) + monkeypatch.setattr(loader, "codex_plugin_roots", lambda: []) + + +def test_plugin_skill_dirs_finds_nested_skill(tmp_path: Path, monkeypatch, _no_external) -> None: + cache = tmp_path / "cache" + root = _install_plugin_with_skill(cache, "ponytail", "ponytail") + monkeypatch.setattr(loader, "plugin_cache_dir", lambda: cache) + + dirs = integration.plugin_skill_dirs() + assert root / "skills" in dirs + + +def test_disabled_plugin_contributes_no_skills(tmp_path: Path, monkeypatch, _no_external) -> None: + cache = tmp_path / "cache" + _install_plugin_with_skill(cache, "ponytail", "ponytail") + monkeypatch.setattr(loader, "plugin_cache_dir", lambda: cache) + + # Enable-set excludes the plugin -> no skill dirs. + assert integration.plugin_skill_dirs(PluginPolicy(enabled=frozenset())) == [] + + +@pytest.mark.asyncio +async def test_resolve_skills_roots_includes_plugin_skill_dir( + tmp_path: Path, monkeypatch, _no_external +) -> None: + from pythinker_code.skill import discover_skills, resolve_skills_roots + + cache = tmp_path / "cache" + root = _install_plugin_with_skill(cache, "ponytail", "ponytail") + monkeypatch.setattr(loader, "plugin_cache_dir", lambda: cache) + # Isolate user/project discovery to an empty home. + monkeypatch.setattr(Path, "home", lambda: tmp_path / "home") + + work_dir = HostPath.unsafe_from_local_path(tmp_path / "project") + roots = await resolve_skills_roots(work_dir) + root_paths = {str(r.root) for r in roots} + assert str((root / "skills").resolve()) in root_paths + + # And the nested skill is actually discoverable from that root. + skills = await discover_skills(HostPath.unsafe_from_local_path(root / "skills"), scope="extra") + assert any(s.name == "ponytail" for s in skills) + + +def _install_plugin_with_agent(cache: Path, plugin: str, agent: str) -> Path: + root = cache / plugin / plugin / "1.0.0" + manifest = root / ".claude-plugin" / "plugin.json" + manifest.parent.mkdir(parents=True, exist_ok=True) + manifest.write_text(json.dumps({"name": plugin, "version": "1.0.0"}), encoding="utf-8") + agent_md = root / "agents" / f"{agent}.md" + agent_md.parent.mkdir(parents=True, exist_ok=True) + agent_md.write_text( + f"---\nname: {agent}\ndescription: {agent} agent\n---\nPrompt body\n", encoding="utf-8" + ) + return root + + +def test_plugin_agent_dirs_finds_agents(tmp_path: Path, monkeypatch, _no_external) -> None: + cache = tmp_path / "cache" + root = _install_plugin_with_agent(cache, "tools", "reviewer") + monkeypatch.setattr(loader, "plugin_cache_dir", lambda: cache) + assert root / "agents" in integration.plugin_agent_dirs() + + +@pytest.mark.asyncio +async def test_resolve_agent_roots_includes_plugin_agents( + tmp_path: Path, monkeypatch, _no_external +) -> None: + from pythinker_code.subagents.discovery import discover_markdown_agents, resolve_agent_roots + + cache = tmp_path / "cache" + _install_plugin_with_agent(cache, "tools", "reviewer") + monkeypatch.setattr(loader, "plugin_cache_dir", lambda: cache) + monkeypatch.setattr(Path, "home", lambda: tmp_path / "home") + + roots = await resolve_agent_roots(HostPath.unsafe_from_local_path(tmp_path / "project")) + assert any(r.scope == "plugin" for r in roots) + agents = await discover_markdown_agents(roots) + assert any(a.name == "reviewer" for a in agents) + + +def _install_plugin_with_command(cache: Path, plugin: str, command: str) -> Path: + root = cache / plugin / plugin / "1.0.0" + manifest = root / ".claude-plugin" / "plugin.json" + manifest.parent.mkdir(parents=True, exist_ok=True) + manifest.write_text(json.dumps({"name": plugin, "version": "1.0.0"}), encoding="utf-8") + cmd_md = root / "commands" / f"{command}.md" + cmd_md.parent.mkdir(parents=True, exist_ok=True) + cmd_md.write_text(f"---\ndescription: {command} command\n---\nDo {command}\n", encoding="utf-8") + return root + + +@pytest.mark.asyncio +async def test_discover_prompt_templates_includes_plugin_commands( + tmp_path: Path, monkeypatch, _no_external +) -> None: + from pythinker_code.prompt_templates import discover_prompt_templates + + cache = tmp_path / "cache" + _install_plugin_with_command(cache, "tools", "ship") + monkeypatch.setattr(loader, "plugin_cache_dir", lambda: cache) + monkeypatch.setattr(Path, "home", lambda: tmp_path / "home") + + templates = await discover_prompt_templates(HostPath.unsafe_from_local_path(tmp_path / "proj")) + assert "ship" in templates + assert templates["ship"].scope == "plugin" + + +def test_plugin_mcp_servers_collects_from_manifest( + tmp_path: Path, monkeypatch, _no_external +) -> None: + cache = tmp_path / "cache" + root = cache / "db" / "db" / "1.0.0" + manifest = root / ".claude-plugin" / "plugin.json" + manifest.parent.mkdir(parents=True) + manifest.write_text( + json.dumps({"name": "db", "version": "1.0.0", "mcpServers": {"pg": {"command": "pg-mcp"}}}), + encoding="utf-8", + ) + monkeypatch.setattr(loader, "plugin_cache_dir", lambda: cache) + + servers = integration.plugin_mcp_servers() + assert servers == {"pg": {"command": "pg-mcp"}} + # Disabled -> contributes nothing. + assert integration.plugin_mcp_servers(PluginPolicy(enabled=frozenset())) == {} + + +def test_plugin_mcp_servers_expands_plugin_root(tmp_path: Path, monkeypatch, _no_external) -> None: + from typing import Any, cast + + cache = tmp_path / "cache" + root = cache / "db" / "db" / "1.0.0" + manifest = root / ".claude-plugin" / "plugin.json" + manifest.parent.mkdir(parents=True) + manifest.write_text( + json.dumps( + { + "name": "db", + "version": "1.0.0", + "mcpServers": {"pg": {"command": "node", "args": ["${CLAUDE_PLUGIN_ROOT}/srv.js"]}}, + } + ), + encoding="utf-8", + ) + monkeypatch.setattr(loader, "plugin_cache_dir", lambda: cache) + + pg = cast("dict[str, Any]", integration.plugin_mcp_servers()["pg"]) + assert pg["args"] == [f"{root}/srv.js"] # ${CLAUDE_PLUGIN_ROOT} expanded to the plugin root + + +def test_plugin_hook_defs_expands_plugin_data(tmp_path: Path, monkeypatch, _no_external) -> None: + from pythinker_code.plugin.directories import plugin_data_dir + + cache = tmp_path / "cache" + root = cache / "sp" / "sp" / "1.0.0" + manifest = root / ".claude-plugin" / "plugin.json" + manifest.parent.mkdir(parents=True) + manifest.write_text(json.dumps({"name": "sp", "version": "1.0.0"}), encoding="utf-8") + hooks = root / "hooks" / "hooks.json" + hooks.parent.mkdir(parents=True) + hooks.write_text( + json.dumps( + { + "hooks": { + "SessionStart": [ + {"hooks": [{"type": "command", "command": "${PYTHINKER_PLUGIN_DATA}/r.sh"}]} + ] + } + } + ), + encoding="utf-8", + ) + monkeypatch.setattr(loader, "plugin_cache_dir", lambda: cache) + + defs = integration.plugin_hook_defs() + assert defs[0].command == f"{plugin_data_dir('sp')}/r.sh" + + +def _mcp_plugin_with_user_config(cache: Path) -> None: + root = cache / "db" / "db" / "1.0.0" + manifest = root / ".claude-plugin" / "plugin.json" + manifest.parent.mkdir(parents=True) + manifest.write_text( + json.dumps( + { + "name": "db", + "version": "1.0.0", + "userConfig": {"token": {"type": "string", "title": "T", "description": "d"}}, + "mcpServers": {"pg": {"command": "pg", "env": {"TOKEN": "${user_config.token}"}}}, + } + ), + encoding="utf-8", + ) + + +def test_plugin_mcp_servers_substitutes_user_config( + tmp_path: Path, monkeypatch, _no_external +) -> None: + from typing import Any, cast + + cache = tmp_path / "cache" + _mcp_plugin_with_user_config(cache) + monkeypatch.setattr(loader, "plugin_cache_dir", lambda: cache) + + servers = integration.plugin_mcp_servers(PluginPolicy(options={"db": {"token": "s3cret"}})) + pg = cast("dict[str, Any]", servers["pg"]) + assert pg["env"]["TOKEN"] == "s3cret" + + +def test_plugin_mcp_servers_skips_unconfigured_user_config( + tmp_path: Path, monkeypatch, _no_external +) -> None: + cache = tmp_path / "cache" + _mcp_plugin_with_user_config(cache) + monkeypatch.setattr(loader, "plugin_cache_dir", lambda: cache) + + # No value configured for ${user_config.token} -> server skipped fail-soft. + assert integration.plugin_mcp_servers() == {} + + +def test_plugin_hook_defs_substitutes_user_config( + tmp_path: Path, monkeypatch, _no_external +) -> None: + cache = tmp_path / "cache" + root = cache / "sp" / "sp" / "1.0.0" + manifest = root / ".claude-plugin" / "plugin.json" + manifest.parent.mkdir(parents=True) + manifest.write_text( + json.dumps( + { + "name": "sp", + "version": "1.0.0", + "userConfig": {"flag": {"type": "string", "title": "F", "description": "d"}}, + "hooks": { + "SessionStart": [ + {"hooks": [{"type": "command", "command": "run ${user_config.flag}"}]} + ] + }, + } + ), + encoding="utf-8", + ) + monkeypatch.setattr(loader, "plugin_cache_dir", lambda: cache) + + defs = integration.plugin_hook_defs(PluginPolicy(options={"sp": {"flag": "on"}})) + assert defs[0].command == "run on" + + # No value configured for ${user_config.flag} -> hook skipped fail-soft, + # mirroring the MCP skip-path so an unresolved placeholder never reaches an + # executable command literally. + assert integration.plugin_hook_defs(PluginPolicy(options={})) == [] + + +def test_plugin_hook_defs_translates_claude_hooks( + tmp_path: Path, monkeypatch, _no_external +) -> None: + cache = tmp_path / "cache" + root = cache / "sp" / "sp" / "1.0.0" + manifest = root / ".claude-plugin" / "plugin.json" + manifest.parent.mkdir(parents=True) + manifest.write_text(json.dumps({"name": "sp", "version": "1.0.0"}), encoding="utf-8") + hooks = root / "hooks" / "hooks.json" + hooks.parent.mkdir(parents=True) + hooks.write_text( + json.dumps( + { + "hooks": { + "SessionStart": [ + { + "matcher": "startup", + "hooks": [ + { + "type": "command", + "command": "${CLAUDE_PLUGIN_ROOT}/run.sh start", + "timeout": 12, + } + ], + } + ], + "BogusEvent": [{"hooks": [{"type": "command", "command": "x"}]}], + } + } + ), + encoding="utf-8", + ) + monkeypatch.setattr(loader, "plugin_cache_dir", lambda: cache) + + defs = integration.plugin_hook_defs() + assert len(defs) == 1 # BogusEvent dropped + hook = defs[0] + assert hook.event == "SessionStart" + assert hook.matcher == "startup" + assert hook.timeout == 12 + # ${CLAUDE_PLUGIN_ROOT} expanded to the plugin root. + assert str(root) in hook.command + assert "${CLAUDE_PLUGIN_ROOT}" not in hook.command + + +def test_plugin_hook_defs_inline_manifest_and_malformed_skip( + tmp_path: Path, monkeypatch, _no_external +) -> None: + cache = tmp_path / "cache" + root = cache / "ip" / "ip" / "1.0.0" + manifest = root / ".claude-plugin" / "plugin.json" + manifest.parent.mkdir(parents=True) + manifest.write_text( + json.dumps( + { + "name": "ip", + "version": "1.0.0", + "hooks": { + "PreToolUse": [ + { + "hooks": [ + {"type": "command", "command": "ok"}, + {"type": "other", "command": "ignored"}, + {"command": 123}, + ] + } + ] + }, + } + ), + encoding="utf-8", + ) + monkeypatch.setattr(loader, "plugin_cache_dir", lambda: cache) + + defs = integration.plugin_hook_defs() + assert [d.command for d in defs] == ["ok"] # non-command + malformed dropped + + +def test_external_skills_auto_detected_but_disablable(tmp_path: Path, monkeypatch) -> None: + # A plugin only in the Claude root: its skills auto-detect by default (no + # symlink, no config), and discover_external=False turns it off. + claude = tmp_path / "claude" + _install_plugin_with_skill(claude, "ponytail", "ponytail") + monkeypatch.setattr(loader, "plugin_cache_dir", lambda: tmp_path / "empty") + monkeypatch.setattr(loader, "claude_plugin_roots", lambda: [claude]) + monkeypatch.setattr(loader, "codex_plugin_roots", lambda: []) + + assert integration.plugin_skill_dirs() # default: auto-detected + assert integration.plugin_skill_dirs(PluginPolicy(discover_external=False)) == [] diff --git a/tests/test_plugin_loader.py b/tests/test_plugin_loader.py new file mode 100644 index 00000000..22923023 --- /dev/null +++ b/tests/test_plugin_loader.py @@ -0,0 +1,167 @@ +"""Tests for plugin discovery, loading, and artifact resolution.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from pythinker_code.plugin import artifacts, loader + + +def _make_plugin( + root: Path, + name: str, + *, + manifest_dir: str = ".claude-plugin", + extra: dict | None = None, +) -> Path: + payload = {"name": name, "version": "1.0.0", **(extra or {})} + manifest_path = root / manifest_dir / "plugin.json" + manifest_path.parent.mkdir(parents=True, exist_ok=True) + manifest_path.write_text(json.dumps(payload), encoding="utf-8") + return root + + +def test_discovers_plugin_nested_like_claude_cache(tmp_path: Path, monkeypatch) -> None: + # Mirror ~/.claude/plugins/cache////.claude-plugin/ + cache = tmp_path / "claude" / "cache" + plugin_root = cache / "ponytail" / "ponytail" / "4.3.0" + _make_plugin(plugin_root, "ponytail") + + monkeypatch.setattr(loader, "plugin_cache_dir", lambda: tmp_path / "none") + monkeypatch.setattr(loader, "claude_plugin_roots", lambda: [cache]) + monkeypatch.setattr(loader, "codex_plugin_roots", lambda: []) + + result = loader.discover_plugins() + names = {p.name for p in result.plugins} + assert "ponytail" in names + found = next(p for p in result.plugins if p.name == "ponytail") + assert found.origin == "claude" + assert found.root == plugin_root + + +def test_malformed_plugin_is_collected_not_raised(tmp_path: Path, monkeypatch) -> None: + cache = tmp_path / "cache" + bad = cache / "bad" + (bad / ".claude-plugin").mkdir(parents=True) + (bad / ".claude-plugin" / "plugin.json").write_text("{broken", encoding="utf-8") + _make_plugin(cache / "good", "good") + + monkeypatch.setattr(loader, "plugin_cache_dir", lambda: cache) + monkeypatch.setattr(loader, "claude_plugin_roots", lambda: []) + monkeypatch.setattr(loader, "codex_plugin_roots", lambda: []) + + result = loader.discover_plugins() + assert {p.name for p in result.plugins} == {"good"} + assert len(result.errors) == 1 + assert "bad" in str(result.errors[0].root) + + +def test_pythinker_origin_wins_over_external_on_name_clash(tmp_path: Path, monkeypatch) -> None: + native = tmp_path / "native" + external = tmp_path / "claude" + _make_plugin(native / "dup", "dup", extra={"description": "native"}) + _make_plugin(external / "dup", "dup", extra={"description": "external"}) + + monkeypatch.setattr(loader, "plugin_cache_dir", lambda: native) + monkeypatch.setattr(loader, "claude_plugin_roots", lambda: [external]) + monkeypatch.setattr(loader, "codex_plugin_roots", lambda: []) + + result = loader.discover_plugins() + dup = [p for p in result.plugins if p.name == "dup"] + assert len(dup) == 1 + assert dup[0].origin == "pythinker" + assert dup[0].manifest.description == "native" + + +def test_include_external_false_skips_claude_codex(tmp_path: Path, monkeypatch) -> None: + _make_plugin(tmp_path / "claude" / "p", "claude-only") + monkeypatch.setattr(loader, "plugin_cache_dir", lambda: tmp_path / "empty") + monkeypatch.setattr(loader, "claude_plugin_roots", lambda: [tmp_path / "claude"]) + monkeypatch.setattr(loader, "codex_plugin_roots", lambda: []) + + result = loader.discover_plugins(include_external=False) + assert result.plugins == [] + + +def test_is_enabled_filter_marks_disabled(tmp_path: Path, monkeypatch) -> None: + _make_plugin(tmp_path / "cache" / "on", "on") + _make_plugin(tmp_path / "cache" / "off", "off") + monkeypatch.setattr(loader, "plugin_cache_dir", lambda: tmp_path / "cache") + monkeypatch.setattr(loader, "claude_plugin_roots", lambda: []) + monkeypatch.setattr(loader, "codex_plugin_roots", lambda: []) + + result = loader.discover_plugins(is_enabled={"on"}) + assert {p.name for p in result.enabled} == {"on"} + assert len(result.plugins) == 2 + + +def _loaded(root: Path, **manifest_extra) -> loader.LoadedPlugin: + _make_plugin(root, "p", extra=manifest_extra) + from pythinker_code.plugin.manifest import load_plugin_manifest + + return loader.LoadedPlugin( + name="p", root=root, manifest=load_plugin_manifest(root), origin="pythinker" + ) + + +def test_skill_dirs_convention_and_override(tmp_path: Path) -> None: + root = tmp_path / "p1" + (root / "skills").mkdir(parents=True) + plugin = _loaded(root) + assert artifacts.skill_dirs(plugin) == [root / "skills"] + + root2 = tmp_path / "p2" + (root2 / "custom").mkdir(parents=True) + plugin2 = _loaded(root2, skills="custom") + assert artifacts.skill_dirs(plugin2) == [(root2 / "custom").resolve()] + + +def test_artifact_path_traversal_is_blocked(tmp_path: Path) -> None: + outside = tmp_path / "outside" + outside.mkdir() + root = tmp_path / "p" + plugin = _loaded(root, skills="../outside") + # Escaping path is rejected -> no dirs contributed. + assert artifacts.skill_dirs(plugin) == [] + + +def test_hooks_file_convention(tmp_path: Path) -> None: + root = tmp_path / "p" + hooks = root / "hooks" / "hooks.json" + hooks.parent.mkdir(parents=True) + hooks.write_text("{}", encoding="utf-8") + plugin = _loaded(root) + assert artifacts.hooks_file(plugin) == hooks + + +def test_mcp_servers_merges_manifest_and_file(tmp_path: Path) -> None: + root = tmp_path / "p" + root.mkdir() + (root / ".mcp.json").write_text( + json.dumps({"mcpServers": {"a": {"command": "x"}, "b": {"command": "y"}}}), + encoding="utf-8", + ) + plugin = _loaded(root, mcpServers={"b": {"command": "override"}, "c": {"command": "z"}}) + servers = artifacts.mcp_servers(plugin) + assert set(servers) == {"a", "b", "c"} + assert servers["b"] == {"command": "override"} # manifest wins + + +def test_mcp_servers_malformed_file_contributes_nothing(tmp_path: Path) -> None: + root = tmp_path / "p" + root.mkdir() + (root / ".mcp.json").write_text("{broken", encoding="utf-8") + plugin = _loaded(root) + assert artifacts.mcp_servers(plugin) == {} + + +@pytest.mark.parametrize("missing_root", ["does/not/exist"]) +def test_discovery_skips_absent_roots(tmp_path: Path, monkeypatch, missing_root: str) -> None: + monkeypatch.setattr(loader, "plugin_cache_dir", lambda: tmp_path / missing_root) + monkeypatch.setattr(loader, "claude_plugin_roots", lambda: []) + monkeypatch.setattr(loader, "codex_plugin_roots", lambda: []) + result = loader.discover_plugins() + assert result.plugins == [] and result.errors == [] diff --git a/tests/test_plugin_manifest.py b/tests/test_plugin_manifest.py new file mode 100644 index 00000000..86e24bac --- /dev/null +++ b/tests/test_plugin_manifest.py @@ -0,0 +1,128 @@ +"""Tests for plugin/marketplace manifest schemas and resolution.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from pythinker_code.plugin.manifest import ( + MANIFEST_DIRS, + MarketplaceManifest, + PluginManifest, + PluginManifestError, + find_plugin_manifest, + load_marketplace_manifest, + load_plugin_manifest, +) + + +def _write(path: Path, payload: dict) -> Path: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(payload), encoding="utf-8") + return path + + +@pytest.mark.parametrize("manifest_dir", [*MANIFEST_DIRS, ""]) +def test_find_plugin_manifest_across_ecosystems(tmp_path: Path, manifest_dir: str) -> None: + root = tmp_path / "plug" + target = root / manifest_dir / "plugin.json" if manifest_dir else root / "plugin.json" + _write(target, {"name": "p", "version": "1.0.0"}) + assert find_plugin_manifest(root) == target + + +def test_manifest_dir_priority_prefers_pythinker(tmp_path: Path) -> None: + root = tmp_path / "plug" + _write(root / ".claude-plugin" / "plugin.json", {"name": "claude", "version": "1"}) + _write(root / ".pythinker-plugin" / "plugin.json", {"name": "pythinker", "version": "1"}) + # .pythinker-plugin wins over .claude-plugin (first in MANIFEST_DIRS). + assert load_plugin_manifest(root).name == "pythinker" + + +def test_claude_minimal_manifest_loads_with_convention_fallback(tmp_path: Path) -> None: + # A real Claude plugin.json carries no artifact paths — artifacts come from + # convention dirs. The manifest must still load with empty overrides. + root = tmp_path / "ponytail" + _write( + root / ".claude-plugin" / "plugin.json", + {"name": "ponytail", "version": "4.3.0", "author": {"name": "x"}}, + ) + manifest = load_plugin_manifest(root) + assert manifest.name == "ponytail" + assert manifest.skills == [] and manifest.commands == [] + + +def test_author_accepts_string_or_object(tmp_path: Path) -> None: + root = tmp_path / "p" + _write(root / "plugin.json", {"name": "p", "version": "1", "author": "Jane"}) + author = load_plugin_manifest(root).author + assert author is not None + assert author.name == "Jane" + + +def test_artifact_paths_normalize_string_and_list() -> None: + m = PluginManifest.model_validate( + {"name": "p", "skills": "skills", "commands": ["a", "b"], "outputStyles": None} + ) + assert m.skills == ["skills"] + assert m.commands == ["a", "b"] + assert m.output_styles == [] + + +def test_mcp_servers_and_dependencies_parse() -> None: + m = PluginManifest.model_validate( + { + "name": "p", + "mcpServers": {"db": {"command": "x"}}, + "dependencies": ["other@market"], + } + ) + assert m.mcp_servers == {"db": {"command": "x"}} + assert m.dependencies == ["other@market"] + + +def test_missing_name_is_typed_error(tmp_path: Path) -> None: + root = tmp_path / "p" + _write(root / "plugin.json", {"version": "1"}) + with pytest.raises(PluginManifestError, match="Missing required field 'name'"): + load_plugin_manifest(root) + + +def test_no_manifest_is_typed_error(tmp_path: Path) -> None: + with pytest.raises(PluginManifestError, match="No plugin.json"): + load_plugin_manifest(tmp_path / "empty") + + +def test_malformed_json_is_typed_error(tmp_path: Path) -> None: + root = tmp_path / "p" + (root).mkdir() + (root / "plugin.json").write_text("{not json", encoding="utf-8") + with pytest.raises(PluginManifestError, match="Failed to read"): + load_plugin_manifest(root) + + +def test_marketplace_manifest_parses_entries(tmp_path: Path) -> None: + path = _write( + tmp_path / ".claude-plugin" / "marketplace.json", + { + "name": "official", + "owner": "OpenAI", + "metadata": {"version": "1.0.2"}, + "plugins": [ + {"name": "codex", "version": "1.0.2", "source": "./plugins/codex"}, + ], + }, + ) + market = load_marketplace_manifest(path) + assert isinstance(market, MarketplaceManifest) + assert market.name == "official" + assert market.owner is not None and market.owner.name == "OpenAI" + assert len(market.plugins) == 1 + assert market.plugins[0].source == "./plugins/codex" + + +def test_marketplace_missing_name_is_typed_error(tmp_path: Path) -> None: + path = _write(tmp_path / "marketplace.json", {"plugins": []}) + with pytest.raises(PluginManifestError, match="Missing required field 'name'"): + load_marketplace_manifest(path) diff --git a/tests/test_plugin_marketplace.py b/tests/test_plugin_marketplace.py new file mode 100644 index 00000000..db8b178b --- /dev/null +++ b/tests/test_plugin_marketplace.py @@ -0,0 +1,159 @@ +"""Tests for marketplace registry, source parsing, and installed registry.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from pythinker_code.plugin import installed, marketplace +from pythinker_code.plugin.marketplace import MarketplaceError, parse_marketplace_input + + +@pytest.fixture(autouse=True) +def _share_dir(tmp_path: Path, monkeypatch): + monkeypatch.setenv("PYTHINKER_SHARE_DIR", str(tmp_path / "share")) + + +# --- parse_marketplace_input ------------------------------------------------- + + +def test_parse_github_shorthand() -> None: + src = parse_marketplace_input("anthropics/claude-plugins-official") + assert src.source == "github" + assert src.repo == "anthropics/claude-plugins-official" + assert src.ref is None + + +def test_parse_github_shorthand_with_ref() -> None: + src = parse_marketplace_input("owner/repo#v2") + assert src.source == "github" and src.repo == "owner/repo" and src.ref == "v2" + + +def test_parse_github_url_becomes_git() -> None: + src = parse_marketplace_input("https://github.com/owner/repo") + assert src.source == "git" and src.url == "https://github.com/owner/repo.git" + + +def test_parse_git_url_with_dot_git() -> None: + src = parse_marketplace_input("https://example.com/x/y.git#main") + assert src.source == "git" and src.ref == "main" + + +def test_parse_ssh_url() -> None: + src = parse_marketplace_input("git@github.com:owner/repo.git") + assert src.source == "git" and src.url == "git@github.com:owner/repo.git" + + +def test_parse_plain_url() -> None: + src = parse_marketplace_input("https://example.com/market.json") + assert src.source == "url" + + +def test_parse_local_directory(tmp_path: Path) -> None: + d = tmp_path / "mk" + d.mkdir() + src = parse_marketplace_input(str(d)) + assert src.source == "directory" and src.path == str(d.resolve()) + + +def test_parse_local_json_file(tmp_path: Path) -> None: + f = tmp_path / "m.json" + f.write_text("{}", encoding="utf-8") + src = parse_marketplace_input(str(f)) + assert src.source == "file" + + +def test_parse_missing_path_raises(tmp_path: Path) -> None: + with pytest.raises(MarketplaceError, match="does not exist"): + parse_marketplace_input(str(tmp_path / "nope")) + + +def test_parse_non_json_file_raises(tmp_path: Path) -> None: + f = tmp_path / "m.txt" + f.write_text("x", encoding="utf-8") + with pytest.raises(MarketplaceError, match=r"must be \.json"): + parse_marketplace_input(str(f)) + + +def test_parse_rejects_plaintext_http() -> None: + with pytest.raises(MarketplaceError, match="https"): + parse_marketplace_input("http://example.test/marketplace.json") + + +def test_parse_unrecognized_raises() -> None: + with pytest.raises(MarketplaceError): + parse_marketplace_input("just-a-word") + + +# --- known_marketplaces registry -------------------------------------------- + + +def test_add_list_remove_marketplace() -> None: + src = parse_marketplace_input("anthropics/claude-plugins-official") + marketplace.add_marketplace("official", src) + + loaded = marketplace.load_known_marketplaces() + assert "official" in loaded + assert loaded["official"].source.repo == "anthropics/claude-plugins-official" + assert loaded["official"].install_location is not None + + assert marketplace.remove_marketplace("official") is True + assert "official" not in marketplace.load_known_marketplaces() + assert marketplace.remove_marketplace("official") is False + + +def test_known_marketplaces_skips_corrupt_entry() -> None: + path = marketplace.known_marketplaces_file() + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text( + json.dumps({"good": {"source": {"source": "github", "repo": "a/b"}}, "bad": 123}), + encoding="utf-8", + ) + loaded = marketplace.load_known_marketplaces() + assert set(loaded) == {"good"} + + +def test_resolve_local_directory_marketplace(tmp_path: Path) -> None: + market_dir = tmp_path / "mk" + manifest = market_dir / ".claude-plugin" / "marketplace.json" + manifest.parent.mkdir(parents=True) + manifest.write_text( + json.dumps({"name": "mk", "plugins": [{"name": "p", "source": "./p"}]}), + encoding="utf-8", + ) + src = parse_marketplace_input(str(market_dir)) + manifest_obj = marketplace.resolve_local_marketplace(src) + assert manifest_obj.name == "mk" + assert manifest_obj.plugins[0].name == "p" + + +# --- installed registry ------------------------------------------------------ + + +def test_record_and_remove_install() -> None: + rec = installed.InstalledRecord(installPath="/x/y", version="1.0.0") + installed.record_install("p", "official", rec) + + loaded = installed.load_installed_plugins() + assert "p@official" in loaded + assert loaded["p@official"][0].install_path == "/x/y" + + assert installed.remove_install("p", "official") is True + assert installed.load_installed_plugins() == {} + + +def test_record_install_replaces_same_scope() -> None: + installed.record_install("p", "m", installed.InstalledRecord(installPath="/a", scope="user")) + installed.record_install("p", "m", installed.InstalledRecord(installPath="/b", scope="user")) + records = installed.load_installed_plugins()["p@m"] + assert len(records) == 1 and records[0].install_path == "/b" + + +def test_remove_install_by_scope() -> None: + installed.record_install("p", "m", installed.InstalledRecord(installPath="/a", scope="user")) + installed.record_install("p", "m", installed.InstalledRecord(installPath="/b", scope="project")) + assert installed.remove_install("p", "m", scope="user") is True + remaining = installed.load_installed_plugins()["p@m"] + assert len(remaining) == 1 and remaining[0].scope == "project" diff --git a/tests/test_plugin_policy.py b/tests/test_plugin_policy.py new file mode 100644 index 00000000..c4f16170 --- /dev/null +++ b/tests/test_plugin_policy.py @@ -0,0 +1,152 @@ +"""Tests for the plugin activation policy and config-driven wiring.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from pythinker_code.plugin import integration, loader +from pythinker_code.plugin.policy import ( + PluginPolicy, + current_plugin_policy, + policy_from_config, + reset_plugin_policy, + set_plugin_policy, +) + + +def test_policy_from_config_defaults() -> None: + pol = policy_from_config(discover_external=True, external_exec=False, enabled=[]) + assert pol.discover_external is True + assert pol.external_exec is False + assert pol.enabled is None # empty list -> all enabled + + +def test_policy_from_config_named_enable_set() -> None: + pol = policy_from_config(discover_external=True, external_exec=True, enabled=["a", "b"]) + assert pol.enabled == frozenset({"a", "b"}) + assert pol.external_exec is True + + +def test_policy_from_config_blank_entries_mean_enable_all() -> None: + # A stray [""] / whitespace-only entry must not silently disable every plugin: + # blanks are dropped and an all-blank list collapses to None (enable all). + def enabled_for(entries: list[str]) -> frozenset[str] | None: + return policy_from_config( + discover_external=True, external_exec=False, enabled=entries + ).enabled + + assert enabled_for([""]) is None + assert enabled_for([" "]) is None + assert enabled_for(["", " a "]) == frozenset({"a"}) # blanks dropped, real names survive + + +def test_policy_from_config_disabled_drops_blanks() -> None: + pol = policy_from_config( + discover_external=True, external_exec=False, enabled=[], disabled=["", " x "] + ) + assert pol.disabled == frozenset({"x"}) + + +def test_default_policy_auto_detects_external_safe_only() -> None: + # The shipped default: external skills/commands/agents auto-detect; exec off. + default = PluginPolicy() + assert default.discover_external is True + assert default.external_exec is False + + +def test_set_and_reset_policy() -> None: + assert current_plugin_policy() == PluginPolicy() # default + token = set_plugin_policy(PluginPolicy(external_exec=True)) + try: + assert current_plugin_policy().external_exec is True + finally: + reset_plugin_policy(token) + assert current_plugin_policy() == PluginPolicy() + + +def _install_external_skill_plugin(claude_root: Path, name: str) -> None: + root = claude_root / name / name / "1.0.0" + pm = root / ".claude-plugin" / "plugin.json" + pm.parent.mkdir(parents=True) + pm.write_text(json.dumps({"name": name, "version": "1.0.0"}), encoding="utf-8") + skill = root / "skills" / name / "SKILL.md" + skill.parent.mkdir(parents=True) + skill.write_text(f"---\nname: {name}\ndescription: d\n---\n# {name}", encoding="utf-8") + + +@pytest.fixture +def _external_ponytail(tmp_path: Path, monkeypatch): + claude = tmp_path / "claude" + _install_external_skill_plugin(claude, "ponytail") + monkeypatch.setattr(loader, "plugin_cache_dir", lambda: tmp_path / "empty") + monkeypatch.setattr(loader, "claude_plugin_roots", lambda: [claude]) + monkeypatch.setattr(loader, "codex_plugin_roots", lambda: []) + + +def test_external_skills_auto_detected_by_default(_external_ponytail) -> None: + # No config, no symlink: an external Claude plugin's skills are found. + assert integration.plugin_skill_dirs() + + +def test_discover_external_false_ignores_external(_external_ponytail) -> None: + token = set_plugin_policy(PluginPolicy(discover_external=False)) + try: + assert integration.plugin_skill_dirs() == [] + finally: + reset_plugin_policy(token) + + +def test_enable_filter_excludes_external(_external_ponytail) -> None: + token = set_plugin_policy( + policy_from_config(discover_external=True, external_exec=False, enabled=["other"]) + ) + try: + assert integration.plugin_skill_dirs() == [] + finally: + reset_plugin_policy(token) + + +def test_disabled_excludes_plugin(_external_ponytail) -> None: + # ponytail auto-detects by default; disabling it by name turns it off. + assert integration.plugin_skill_dirs() + token = set_plugin_policy(PluginPolicy(disabled=frozenset({"ponytail"}))) + try: + assert integration.plugin_skill_dirs() == [] + finally: + reset_plugin_policy(token) + + +def _install_external_mcp_plugin(claude_root: Path, name: str) -> None: + root = claude_root / name / name / "1.0.0" + pm = root / ".claude-plugin" / "plugin.json" + pm.parent.mkdir(parents=True) + pm.write_text( + json.dumps({"name": name, "version": "1.0.0", "mcpServers": {"db": {"command": "x"}}}), + encoding="utf-8", + ) + + +def test_external_mcp_is_opt_in(tmp_path: Path, monkeypatch) -> None: + claude = tmp_path / "claude" + _install_external_mcp_plugin(claude, "dbplug") + monkeypatch.setattr(loader, "plugin_cache_dir", lambda: tmp_path / "empty") + monkeypatch.setattr(loader, "claude_plugin_roots", lambda: [claude]) + monkeypatch.setattr(loader, "codex_plugin_roots", lambda: []) + + # Default: external exec artifacts (MCP) are NOT activated. + assert integration.plugin_mcp_servers() == {} + # Opt-in via external_exec. + token = set_plugin_policy(PluginPolicy(external_exec=True)) + try: + assert integration.plugin_mcp_servers() == {"db": {"command": "x"}} + finally: + reset_plugin_policy(token) + + +@pytest.fixture(autouse=True) +def _reset_policy(): + yield + set_plugin_policy(PluginPolicy()) diff --git a/tests/tools/test_mcp_resource.py b/tests/tools/test_mcp_resource.py index 97897efa..4b100ec1 100644 --- a/tests/tools/test_mcp_resource.py +++ b/tests/tools/test_mcp_resource.py @@ -5,7 +5,7 @@ from typing import Any from pythinker_code.soul.toolset import MCPServerInfo, PythinkerToolset -from pythinker_code.tools.mcp_resource import ListMcpResources, ReadMcpResource +from pythinker_code.tools.mcp_resource import InvokeMcpPrompt, ListMcpResources, ReadMcpResource async def test_discover_optional_capability_distinguishes_absent_from_transient() -> None: @@ -82,6 +82,7 @@ def __init__( self._contents = contents or [] self._raise = raise_on_read self._raise_on_enter = raise_on_enter + self.prompt_calls: list[tuple[str, dict[str, Any]]] = [] async def __aenter__(self) -> _FakeClient: if self._raise_on_enter: @@ -96,6 +97,16 @@ async def read_resource(self, uri: str) -> list[Any]: raise RuntimeError("boom") return self._contents + async def get_prompt(self, name: str, arguments: dict[str, Any]) -> Any: + if self._raise: + raise RuntimeError("boom") + self.prompt_calls.append((name, arguments)) + return type( + "PromptResult", + (), + {"messages": [type("PromptMessage", (), {"role": "user", "content": "summarize"})()]}, + )() + def _toolset_with_server(name: str, **kw: Any) -> PythinkerToolset: ts = PythinkerToolset() @@ -171,3 +182,30 @@ async def test_read_resource_from_unconnectable_server_errors() -> None: result = await ReadMcpResource(ts)(ReadMcpResource.params(server="db", uri="x")) assert result.is_error assert result.brief == "Resource read failed" + + +async def test_invoke_prompt_returns_untrusted_messages() -> None: + client = _FakeClient() + ts = _toolset_with_server("db", client=client, prompts=[_Prompt("summarize")]) + + result = await InvokeMcpPrompt(ts)( + InvokeMcpPrompt.params(server="db", name="summarize", arguments={"table": "users"}) + ) + + assert not result.is_error + assert client.prompt_calls == [("summarize", {"table": "users"})] + assert isinstance(result.output, str) + assert "role: user" in result.output + assert "summarize" in result.output + assert "untrusted_data" in result.output + + +async def test_invoke_prompt_unknown_prompt_errors() -> None: + ts = _toolset_with_server("db", prompts=[_Prompt("summarize")]) + + result = await InvokeMcpPrompt(ts)( + InvokeMcpPrompt.params(server="db", name="missing", arguments={}) + ) + + assert result.is_error + assert result.brief == "Unknown MCP prompt" diff --git a/tests/tools/test_recall.py b/tests/tools/test_recall.py index 8079f121..335acd51 100644 --- a/tests/tools/test_recall.py +++ b/tests/tools/test_recall.py @@ -19,7 +19,7 @@ def _session(sid: str, *, title: str = "", custom_title: str = "", updated_at: f id=sid, title=title, updated_at=updated_at, - state=SimpleNamespace(custom_title=custom_title), + state=SimpleNamespace(custom_title=custom_title, plan_slug=None), ) @@ -28,6 +28,22 @@ def _ranked_ids(sessions: list[Any], *, query: str, current_id: str) -> list[str return [s.id for s in ranked] +def test_rank_matches_session_id_and_plan_slug() -> None: + sessions = [ + _session("sess-auth-1", custom_title="misc", updated_at=1.0), + _session( + "other", + custom_title="plan work", + updated_at=2.0, + ), + ] + sessions[1].state.plan_slug = "auth-migration" + ids = _ranked_ids(sessions, query="auth-migration", current_id="cur") + assert ids == ["other"] + ids_by_id = _ranked_ids(sessions, query="sess-auth", current_id="cur") + assert ids_by_id == ["sess-auth-1"] + + def test_rank_excludes_current_and_filters_non_matches() -> None: sessions = [ _session("a", custom_title="auth migration", updated_at=1.0), @@ -118,6 +134,28 @@ def test_render_transcript_budget_truncates(tmp_path: Path) -> None: assert len(rendered) < 700 +def test_render_transcript_supports_message_window(tmp_path: Path) -> None: + log = tmp_path / "context.jsonl" + log.write_text( + "\n".join( + [ + '{"role": "_checkpoint", "content": [{"type": "text", "text": "internal"}]}', + Message(role="user", content=[TextPart(text="first")]).model_dump_json(), + Message(role="assistant", content=[TextPart(text="second")]).model_dump_json(), + Message(role="user", content=[TextPart(text="third")]).model_dump_json(), + ] + ), + encoding="utf-8", + ) + + rendered = _render_transcript(log, budget=10_000, message_offset=1, max_messages=1) + + assert "[assistant] second" in rendered + assert "first" not in rendered + assert "third" not in rendered + assert "internal" not in rendered + + async def test_recall_search_lists_matching_sessions( runtime, monkeypatch: pytest.MonkeyPatch ) -> None: @@ -136,6 +174,22 @@ async def fake_list(work_dir: Any) -> list[Any]: assert "s2" not in result.output +async def test_recall_search_includes_plan_slug_when_present( + runtime, monkeypatch: pytest.MonkeyPatch +) -> None: + async def fake_list(work_dir: Any) -> list[Any]: + session = _session("s1", custom_title="plan work", updated_at=2.0) + session.state.plan_slug = "auth-migration" + return [session] + + monkeypatch.setattr(Session, "list", staticmethod(fake_list)) + result = await Recall(runtime)(Recall.params(mode="search", query="auth-migration")) + + assert not result.is_error + assert isinstance(result.output, str) + assert "plan_slug: auth-migration" in result.output + + async def test_recall_read_returns_untrusted_transcript( runtime, tmp_path: Path, monkeypatch: pytest.MonkeyPatch ) -> None: @@ -155,6 +209,33 @@ async def fake_find(work_dir: Any, session_id: str) -> Any: assert "untrusted_data" in result.output +async def test_recall_read_passes_message_window( + runtime, tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + log = tmp_path / "context.jsonl" + _write_log( + log, + [ + Message(role="user", content=[TextPart(text="first")]), + Message(role="assistant", content=[TextPart(text="second")]), + ], + ) + fake = SimpleNamespace(id="s1", context_file=log) + + async def fake_find(work_dir: Any, session_id: str) -> Any: + return fake if session_id == "s1" else None + + monkeypatch.setattr(Session, "find", staticmethod(fake_find)) + result = await Recall(runtime)( + Recall.params(mode="read", session_id="s1", message_offset=1, max_messages=1) + ) + + assert not result.is_error + assert isinstance(result.output, str) + assert "second" in result.output + assert "first" not in result.output + + async def test_recall_read_unknown_session_errors(runtime, monkeypatch: pytest.MonkeyPatch) -> None: async def fake_find(work_dir: Any, session_id: str) -> Any: return None diff --git a/tests/tools/test_suggest.py b/tests/tools/test_suggest.py index f36170db..50bb08f3 100644 --- a/tests/tools/test_suggest.py +++ b/tests/tools/test_suggest.py @@ -45,8 +45,27 @@ async def test_suggest_defaults_blank_prefill_and_category(monkeypatch) -> None: def test_suggestion_block_renders_label_and_prefill() -> None: + from pythinker_code.ui.shell.components import render_plain from pythinker_code.ui.shell.visualize._blocks import _SuggestionBlock block = _SuggestionBlock(Suggestion(label="Review my changes", prefill="/review")) - # compose() must build a renderable without error. - assert block.compose() is not None + rendered = render_plain(block.compose(), width=100) + assert "Review my changes" in rendered + assert "/review" in rendered + assert "Alt+S to accept" in rendered + + +def test_accept_staged_suggestion_prefill_inserts_text() -> None: + from unittest.mock import MagicMock + + from pythinker_code.ui.shell.prompt import CustomPromptSession + + session = CustomPromptSession.__new__(CustomPromptSession) + buffer = MagicMock() + buffer.text = "" + session._session = MagicMock(default_buffer=buffer) + session.stage_suggestion_prefill("/review") + + assert session.accept_staged_suggestion_prefill() is True + buffer.insert_text.assert_called_once_with("/review") + assert session.accept_staged_suggestion_prefill() is False diff --git a/tests/tools/test_tool_descriptions.py b/tests/tools/test_tool_descriptions.py index 87d27ec0..fc1f1973 100644 --- a/tests/tools/test_tool_descriptions.py +++ b/tests/tools/test_tool_descriptions.py @@ -15,6 +15,7 @@ from pythinker_code.tools.file.read_media import ReadMediaFile from pythinker_code.tools.file.replace import StrReplaceFile from pythinker_code.tools.file.write import WriteFile +from pythinker_code.tools.plan import ExitPlanMode from pythinker_code.tools.shell import Shell from pythinker_code.tools.think import Think from pythinker_code.tools.todo import SetTodoList @@ -22,6 +23,13 @@ from pythinker_code.tools.web.search import SearchWeb +def test_exit_plan_mode_description_requires_verification_section(): + tool = ExitPlanMode() + + assert "Verification section" in tool.base.description + assert "smallest command, test, or check" in tool.base.description + + def test_agent_description(agent_tool: AgentTool): """Test the description of Agent tool.""" assert agent_tool.base.description == snapshot( diff --git a/tests/ui_and_conv/_md_contract_helpers.py b/tests/ui_and_conv/_md_contract_helpers.py index c6c71e22..26669645 100644 --- a/tests/ui_and_conv/_md_contract_helpers.py +++ b/tests/ui_and_conv/_md_contract_helpers.py @@ -14,6 +14,8 @@ from rich.console import Console, RenderableType +from pythinker_code.ui.shell.components.render_utils import sanitize_ansi + # Widths that exercise reflow boundaries: very narrow, a normal width, and an # exactly-typical report width. Add the exact-full-width case per test. WIDTHS: tuple[int, ...] = (24, 40, 80) @@ -25,7 +27,7 @@ def render_plain(renderable: RenderableType, *, width: int = 80) -> str: console = Console(width=width, no_color=True, legacy_windows=False) with console.capture() as cap: console.print(renderable) - return cap.get() + return sanitize_ansi(cap.get()) def render_ansi(renderable: RenderableType, *, width: int = 80) -> str: diff --git a/tests/ui_and_conv/test_compaction_block.py b/tests/ui_and_conv/test_compaction_block.py index 470abd67..b0a54196 100644 --- a/tests/ui_and_conv/test_compaction_block.py +++ b/tests/ui_and_conv/test_compaction_block.py @@ -1,3 +1,5 @@ +from rich.text import Text + from pythinker_code.ui.shell.components import render_plain from pythinker_code.ui.shell.visualize._blocks import _CompactionBlock @@ -21,3 +23,14 @@ def test_compaction_block_context_tokens_can_update(): rendered = render_plain(block._render(), width=100) assert "↑ 12.3k tokens" in rendered + + +def test_compaction_block_shows_todos_when_provided(): + todos_text = Text(" ⎿ ■ Refactor auth\n ✓ Write tests") + block = _CompactionBlock(context_tokens=None, todos_renderable=todos_text) + + rendered = render_plain(block._render(), width=100) + + assert "Refactor auth" in rendered + assert "Write tests" in rendered + assert "Tip:" not in rendered diff --git a/tests/ui_and_conv/test_file_completer.py b/tests/ui_and_conv/test_file_completer.py index 5b55dba4..a3908675 100644 --- a/tests/ui_and_conv/test_file_completer.py +++ b/tests/ui_and_conv/test_file_completer.py @@ -148,6 +148,20 @@ def test_basename_prefix_is_ranked_first(tmp_path: Path): ) +def test_test_paths_are_ranked_after_source_matches(tmp_path: Path): + """Prefer source files over equally relevant test paths.""" + (tmp_path / "tests").mkdir() + (tmp_path / "zzz_src").mkdir() + (tmp_path / "tests" / "foo.py").write_text("# test\n") + (tmp_path / "zzz_src" / "foo.py").write_text("# source\n") + + completer = LocalFileMentionCompleter(tmp_path) + + texts = _completion_texts(completer, "@foo") + + assert texts[:2] == ["zzz_src/foo.py", "tests/foo.py"] + + def _init_git_repo(work_dir: Path) -> None: """Initialise a git repo, stage all files, and commit.""" for cmd in ( diff --git a/tests/ui_and_conv/test_live_view_todos.py b/tests/ui_and_conv/test_live_view_todos.py index 0111c85d..2897667a 100644 --- a/tests/ui_and_conv/test_live_view_todos.py +++ b/tests/ui_and_conv/test_live_view_todos.py @@ -2,12 +2,14 @@ import importlib +import pytest from pythinker_core.message import ToolCall from pythinker_core.tooling import ToolResult, ToolReturnValue from rich.color import Color from rich.console import Console, Group from rich.style import Style +from pythinker_code.soul.live_tokens import add_total_output_tokens, reset_for_tests from pythinker_code.tools.display import DiffDisplayBlock, TodoDisplayBlock, TodoDisplayItem from pythinker_code.ui.shell.motion import ( _SHIMMER_BASE, @@ -20,6 +22,15 @@ _live_view_module = importlib.import_module("pythinker_code.ui.shell.visualize._live_view") + +@pytest.fixture(autouse=True) +def _reset_live_tokens(): + """Isolate the session-wide live token accumulator between tests.""" + reset_for_tests() + yield + reset_for_tests() + + _SHIMMER_HEXES = {_SHIMMER_BASE.lower(), _SHIMMER_MID.lower(), _SHIMMER_HIGHLIGHT.lower()} @@ -95,8 +106,9 @@ def test_todo_update_pins_current_task_under_activity_line(monkeypatch) -> None: # Pin the animated braille marker to its static dot for a deterministic # assertion on the activity-line content. monkeypatch.setenv("PYTHINKER_REDUCED_MOTION", "1") - view = _LiveView(StatusUpdate(context_tokens=10_000)) + view = _LiveView(StatusUpdate()) view.dispatch_wire_message(TurnBegin(user_input="work")) + add_total_output_tokens(10_000) # output produced this turn (main + subagents) view.dispatch_wire_message(_todo_call()) view.dispatch_wire_message(_todo_result()) @@ -117,8 +129,9 @@ def test_active_todo_activity_line_does_not_alternate_with_spinner_verb(monkeypa now = 1000.0 monkeypatch.setattr(_live_view_module.time, "monotonic", lambda: now) monkeypatch.setenv("PYTHINKER_REDUCED_MOTION", "1") - view = _LiveView(StatusUpdate(context_tokens=10_000)) + view = _LiveView(StatusUpdate()) view.dispatch_wire_message(TurnBegin(user_input="work")) + add_total_output_tokens(10_000) # output produced this turn (main + subagents) view.dispatch_wire_message(_todo_call()) view.dispatch_wire_message(_todo_result()) @@ -135,8 +148,9 @@ def test_spinner_verb_shows_until_next_todo_becomes_active(monkeypatch) -> None: now = 1000.0 monkeypatch.setattr(_live_view_module.time, "monotonic", lambda: now) monkeypatch.setenv("PYTHINKER_REDUCED_MOTION", "1") - view = _LiveView(StatusUpdate(context_tokens=10_000)) + view = _LiveView(StatusUpdate()) view.dispatch_wire_message(TurnBegin(user_input="work")) + add_total_output_tokens(10_000) # output produced this turn (main + subagents) view._latest_todos = ( TodoDisplayItem(title="Finished task", status="done"), TodoDisplayItem(title="Next task", status="pending"), diff --git a/tests/ui_and_conv/test_markdown_guards.py b/tests/ui_and_conv/test_markdown_guards.py index 5bcb59ad..c3312be6 100644 --- a/tests/ui_and_conv/test_markdown_guards.py +++ b/tests/ui_and_conv/test_markdown_guards.py @@ -4,6 +4,7 @@ from pythinker_code.ui.shell.components.markdown import ( PythinkerMarkdown, + _loosen_tight_ordered_lists, _unwrap_fenced_markdown_tables, pythinker_markdown, ) @@ -114,3 +115,43 @@ def test_small_code_block_still_highlights_without_notice() -> None: out = render_plain(pythinker_markdown("```python\nx = 1\n```"), width=80) assert "highlighting skipped" not in out assert "x = 1" in out + + +# --------------------------------------------------------------------------- +# _loosen_tight_ordered_lists +# --------------------------------------------------------------------------- + + +def test_tight_ol_gets_blank_lines_between_items() -> None: + text = "1. first\n2. second\n3. third\n" + result = _loosen_tight_ordered_lists(text) + assert result == "1. first\n\n2. second\n\n3. third\n" + + +def test_already_loose_ol_unchanged() -> None: + text = "1. first\n\n2. second\n" + assert _loosen_tight_ordered_lists(text) == text + + +def test_single_ol_item_unchanged() -> None: + text = "1. only item\n" + assert _loosen_tight_ordered_lists(text) == text + + +def test_ol_inside_fence_not_loosened() -> None: + text = "```\n1. inside fence\n2. still inside\n```\n1. outside\n2. outside too\n" + result = _loosen_tight_ordered_lists(text) + # Items inside the fence must stay tight; items outside get the blank line + assert "1. inside fence\n2. still inside" in result + assert "1. outside\n\n2. outside too" in result + + +def test_unordered_list_not_affected() -> None: + text = "- bullet one\n- bullet two\n" + assert _loosen_tight_ordered_lists(text) == text + + +def test_ol_mixed_with_prose_inserts_only_between_items() -> None: + text = "Intro.\n1. first\n2. second\nOutro.\n" + result = _loosen_tight_ordered_lists(text) + assert result == "Intro.\n1. first\n\n2. second\nOutro.\n" diff --git a/tests/ui_and_conv/test_md_repair_characterization.py b/tests/ui_and_conv/test_md_repair_characterization.py index 4683940c..021ed01a 100644 --- a/tests/ui_and_conv/test_md_repair_characterization.py +++ b/tests/ui_and_conv/test_md_repair_characterization.py @@ -11,6 +11,7 @@ from __future__ import annotations from pythinker_code.ui.shell.components.markdown import ( + _loosen_tight_ordered_lists, _normalize_markdown_tables, _repair_crammed_markdown_tables, pythinker_markdown, @@ -46,3 +47,39 @@ def test_wellformed_table_is_passed_through_unchanged_in_render(): out = render_plain(pythinker_markdown(clean), width=40) for token in ("A", "B", "1", "2", "3", "4"): assert token in out + + +# --------------------------------------------------------------------------- +# _loosen_tight_ordered_lists — behavior specs (not pinned characterization) +# --------------------------------------------------------------------------- + + +def test_loosen_tight_ol_inserts_blank_between_items(): + inp = "1. First\n2. Second\n3. Third\n" + out = _loosen_tight_ordered_lists(inp) + assert out == "1. First\n\n2. Second\n\n3. Third\n" + + +def test_loosen_tight_ol_skips_already_spaced(): + inp = "1. First\n\n2. Second\n" + out = _loosen_tight_ordered_lists(inp) + assert out == "1. First\n\n2. Second\n" + + +def test_loosen_tight_ol_ignores_inside_fence(): + inp = "```\n1. inside\n2. fence\n```\n" + out = _loosen_tight_ordered_lists(inp) + assert out == "```\n1. inside\n2. fence\n```\n" + + +def test_loosen_tight_ol_preserves_surrounding_text(): + inp = "intro\n1. First\n2. Second\noutro\n" + out = _loosen_tight_ordered_lists(inp) + assert out == "intro\n1. First\n\n2. Second\noutro\n" + + +def test_loosen_tight_ol_unordered_list_unchanged(): + """Unordered list items are not loosened — function is OL-only.""" + inp = "- a\n- b\n- c\n" + out = _loosen_tight_ordered_lists(inp) + assert out == "- a\n- b\n- c\n" diff --git a/tests/ui_and_conv/test_report.py b/tests/ui_and_conv/test_report.py index 821b42d7..edf00e11 100644 --- a/tests/ui_and_conv/test_report.py +++ b/tests/ui_and_conv/test_report.py @@ -5,6 +5,7 @@ import pytest from rich.console import Console +from pythinker_code.ui.shell.components.render_utils import sanitize_ansi from pythinker_code.ui.shell.components.report import ( Report, ReportFinding, @@ -18,7 +19,7 @@ def _plain(renderable, *, width: int = 80) -> str: console = Console(width=width, no_color=True, legacy_windows=False) with console.capture() as cap: console.print(renderable) - return cap.get() + return sanitize_ansi(cap.get()) # --------------------------------------------------------------------------- @@ -100,6 +101,36 @@ def test_render_report_hanging_indents_wrapped_locations(): assert location_lines[0].index("packages") == location_lines[1].index("packages") +def test_render_report_hang_indents_wrapped_finding_title(): + """A wrapped finding title aligns with the body/location column, not back + under the ● marker — so a long title never reads as a separate finding.""" + report = Report( + title="Deep Code Scan Results", + findings=( + ReportFinding( + "disconnect_mcp_server swallows close errors and overwrites info.error on timeout", + "high", + location="src/pythinker_code/soul/toolset.py:1378-1385", + body="The bare except swallows the timeout.", + ), + ), + ) + + out = _plain(render_report(report), width=60) + lines = out.splitlines() + marker_line = next(line for line in lines if "● disconnect_mcp_server" in line) + wrap_line = next(line for line in lines if "overwrites" in line and "●" not in line) + location_line = next(line for line in lines if "toolset.py" in line) + + marker_col = marker_line.index("●") + title_col = marker_line.index("disconnect_mcp_server") + # The wrapped title and the location start at the title column, both deeper + # than the marker — an unambiguous, cohesive finding block. + assert wrap_line.index("overwrites") == title_col + assert location_line.index("src/pythinker") == title_col + assert title_col > marker_col + + def test_render_report_groups_in_severity_order_regardless_of_input(): report = Report( title="t", diff --git a/tests/ui_and_conv/test_review_findings_parser.py b/tests/ui_and_conv/test_review_findings_parser.py index bc75871b..adf688b7 100644 --- a/tests/ui_and_conv/test_review_findings_parser.py +++ b/tests/ui_and_conv/test_review_findings_parser.py @@ -251,3 +251,82 @@ def test_aggregate_empty_result_text_is_unparsed(): assert summary.unparsed_reports == 1 assert summary.parsed_reports == 0 assert "empty_scan" in summary.reporters["unknown"] + + +# --------------------------------------------------------------------------- +# _parse_reviewer_findings — ```report JSON block (primary machine-readable format) +# --------------------------------------------------------------------------- + + +def test_report_block_counts_findings(): + text = ( + "Some analysis.\n" + "```report\n" + '{"findings": [\n' + ' {"title": "Bug A", "severity": "high"},\n' + ' {"title": "Bug B", "severity": "medium"},\n' + ' {"title": "Bug C", "severity": "high"}\n' + "]}\n" + "```\n" + ) + counts, was_parsed = _parse_reviewer_findings(text) + assert was_parsed is True + assert counts == {"critical": 0, "high": 2, "medium": 1, "low": 0} + + +def test_report_block_empty_findings_is_parsed(): + """{"findings": []} is a valid structured report — was_parsed must be True.""" + text = '```report\n{"findings": []}\n```\n' + counts, was_parsed = _parse_reviewer_findings(text) + assert was_parsed is True + assert counts == {"critical": 0, "high": 0, "medium": 0, "low": 0} + + +def test_report_block_malformed_json_is_not_parsed(): + """A malformed ```report block must NOT report success — it is unparsed, so the + caller treats it as an unparsed reviewer result rather than 'parsed, 0 findings'.""" + text = "```report\nnot valid json\n```\n" + counts, was_parsed = _parse_reviewer_findings(text) + assert was_parsed is False + assert counts == {"critical": 0, "high": 0, "medium": 0, "low": 0} + + +def test_report_block_wrong_shape_is_not_parsed(): + """A ```report block holding a JSON array (not an object) is not a valid report.""" + text = '```report\n[{"severity": "high"}]\n```\n' + counts, was_parsed = _parse_reviewer_findings(text) + assert was_parsed is False + assert counts["high"] == 0 + + +def test_report_block_non_dict_findings_does_not_crash(): + """A payload whose 'findings' isn't a list must not raise and isn't 'parsed'.""" + text = '```report\n{"findings": "high"}\n```\n' + counts, was_parsed = _parse_reviewer_findings(text) + # 'findings' is the wrong shape (str, not list) -> not a usable structured + # report, so was_parsed is False (the caller falls back to markdown scanning) + # rather than reporting a false "parsed with zero findings". + assert was_parsed is False + assert counts == {"critical": 0, "high": 0, "medium": 0, "low": 0} + + +def test_report_block_multiple_blocks_aggregate(): + block1 = '```report\n{"findings": [{"severity": "critical"}]}\n```\n' + block2 = '```report\n{"findings": [{"severity": "low"}, {"severity": "low"}]}\n```\n' + counts, was_parsed = _parse_reviewer_findings(block1 + block2) + assert was_parsed is True + assert counts == {"critical": 1, "high": 0, "medium": 0, "low": 2} + + +def test_report_block_with_markdown_markers_uses_json_only(): + """JSON block takes priority — markdown markers inside JSON must not be double-counted.""" + text = ( + '```report\n{"findings": [{"severity": "high", "description": "- [HIGH] inside JSON"}]}\n```\n' + "- [HIGH] outside block\n" + ) + # Only the JSON findings count; the markdown marker outside the block is not scanned + # because the report block path returns early without running the markdown fallback. + counts, was_parsed = _parse_reviewer_findings(text) + assert was_parsed is True + # JSON block found: 1 high from JSON. Markdown fallback is skipped entirely. + assert counts["high"] == 1 diff --git a/tests/ui_and_conv/test_shell_slash_commands.py b/tests/ui_and_conv/test_shell_slash_commands.py index ddc15bff..034c876a 100644 --- a/tests/ui_and_conv/test_shell_slash_commands.py +++ b/tests/ui_and_conv/test_shell_slash_commands.py @@ -249,7 +249,9 @@ def update(self, *args: Any, **kwargs: Any) -> None: assert cmd is not None await _invoke_slash_command(cmd, SimpleNamespace()) - output = capsys.readouterr().out + from pythinker_code.ui.shell.components.render_utils import sanitize_ansi + + output = sanitize_ansi(capsys.readouterr().out) assert live_transient_values == [True] assert "🔌 MCP Tools" in output assert "context7" in output diff --git a/tests/ui_and_conv/test_tui_transcript_enhancements.py b/tests/ui_and_conv/test_tui_transcript_enhancements.py index 905163d6..01a1c8c3 100644 --- a/tests/ui_and_conv/test_tui_transcript_enhancements.py +++ b/tests/ui_and_conv/test_tui_transcript_enhancements.py @@ -27,6 +27,15 @@ _live_view_mod = importlib.import_module("pythinker_code.ui.shell.visualize._live_view") +@pytest.fixture(autouse=True) +def _reset_live_tokens(): + from pythinker_code.soul.live_tokens import reset_for_tests + + reset_for_tests() + yield + reset_for_tests() + + @pytest.fixture(autouse=True) def _builtin_renderers(): clear_tool_renderers() @@ -117,12 +126,21 @@ def test_progress_note_block_strips_ansi_from_title() -> None: assert "Checkpoint" in rendered -def test_working_indicator_includes_context_token_count(monkeypatch) -> None: - monkeypatch.setattr(_live_view_mod.time, "monotonic", lambda: 10.0) - view = _LiveView(StatusUpdate(context_tokens=110_800)) - view.dispatch_wire_message(TurnBegin(user_input="work")) +def test_working_indicator_includes_turn_output_tokens(monkeypatch) -> None: + from pythinker_code.soul.live_tokens import add_total_output_tokens, reset_for_tests + + reset_for_tests() + try: + monkeypatch.setattr(_live_view_mod.time, "monotonic", lambda: 10.0) + view = _LiveView(StatusUpdate()) + view.dispatch_wire_message(TurnBegin(user_input="work")) + # Output produced this turn (main agent + any subagents) — the live + # readout tracks throughput, not the static context-window size. + add_total_output_tokens(110_800) - rendered = render_plain(view._working_indicator(), width=100) + rendered = render_plain(view._working_indicator(), width=100) - assert "110.8k" in rendered - assert "tokens" in rendered + assert "110.8k" in rendered + assert "tokens" in rendered + finally: + reset_for_tests() diff --git a/tests/ui_and_conv/test_visualize_running_prompt.py b/tests/ui_and_conv/test_visualize_running_prompt.py index 89e83a8a..3e69803b 100644 --- a/tests/ui_and_conv/test_visualize_running_prompt.py +++ b/tests/ui_and_conv/test_visualize_running_prompt.py @@ -1614,27 +1614,34 @@ def test_background_status_shows_elapsed_tokens_and_rate(monkeypatch) -> None: """The line above the input carries (elapsed, ↓ tokens, t/s) — the same metadata design as the live view's working indicator.""" import pythinker_code.ui.shell.prompt as prompt_module - from pythinker_code.soul import StatusSnapshot + from pythinker_code.soul import live_tokens + live_tokens.reset_for_tests() session = object.__new__(CustomPromptSession) session._background_task_count_provider = lambda: BgTaskCounts(agent=2) session._latest_todos = () - state = {"now": 100.0, "tokens": 40_000} - session._status_provider = lambda: StatusSnapshot( - context_usage=0.0, context_tokens=state["tokens"] - ) + state = {"now": 100.0, "output_tokens": 0} monkeypatch.setattr(prompt_module.time, "monotonic", lambda: state["now"]) + monkeypatch.setattr( + live_tokens, + "get_total_output_tokens", + lambda: state["output_tokens"], + ) def render() -> str: rendered = CustomPromptSession._render_background_working_status(session, 120) return "".join(item[1] for item in rendered) first = render() - assert "(<1s, ↓ 40k tokens)" in first # no rate until the window fills + assert "(<1s" in first # stretch just started — no output delta yet + + state["now"], state["output_tokens"] = 100.0, 40_000 + second = render() + assert "(<1s, ↓ 40k tokens)" in second # no rate until the window fills - state["now"], state["tokens"] = 100.4, 40_400 + state["now"], state["output_tokens"] = 100.4, 40_400 render() - state["now"], state["tokens"] = 100.8, 40_800 + state["now"], state["output_tokens"] = 100.8, 40_800 third = render() assert "(<1s, ↓ 40.8k tokens, 1000 t/s)" in third @@ -1642,6 +1649,7 @@ def render() -> str: session._background_task_count_provider = lambda: BgTaskCounts() assert render() == "" assert session._bg_status_started_at is None + assert session._bg_status_start_tokens is None # --------------------------------------------------------------------------- diff --git a/tests/utils/test_media_limits.py b/tests/utils/test_media_limits.py new file mode 100644 index 00000000..9a383999 --- /dev/null +++ b/tests/utils/test_media_limits.py @@ -0,0 +1,20 @@ +"""Tests for provider-safe media limits (task 7.5).""" + +from __future__ import annotations + +from pythinker_code.utils.media_limits import ( + MAX_IMAGE_BYTES, + MAX_IMAGE_PIXELS, + MAX_VIDEO_BYTES, + format_byte_limit, +) + + +def test_byte_limits_are_positive_and_ordered() -> None: + assert MAX_IMAGE_BYTES > 0 + assert MAX_VIDEO_BYTES >= MAX_IMAGE_BYTES + assert MAX_IMAGE_PIXELS > 0 + + +def test_format_byte_limit_uses_megabytes_for_large_values() -> None: + assert format_byte_limit(MAX_IMAGE_BYTES) == "20 MB" diff --git a/tests/utils/test_pyinstaller_utils.py b/tests/utils/test_pyinstaller_utils.py index 7e8071f1..a7a62471 100644 --- a/tests/utils/test_pyinstaller_utils.py +++ b/tests/utils/test_pyinstaller_utils.py @@ -223,6 +223,10 @@ def test_pyinstaller_datas(): "src/pythinker_code/tools/mcp_resource/list_description.md", "pythinker_code/tools/mcp_resource", ), + ( + "src/pythinker_code/tools/mcp_resource/prompt_description.md", + "pythinker_code/tools/mcp_resource", + ), ( "src/pythinker_code/tools/mcp_resource/read_description.md", "pythinker_code/tools/mcp_resource", diff --git a/tests_ai/eval_cases.example.json b/tests_ai/eval_cases.example.json new file mode 100644 index 00000000..76538a89 --- /dev/null +++ b/tests_ai/eval_cases.example.json @@ -0,0 +1,11 @@ +[ + { + "name": "encoding smoke", + "query": "verify utf-8 handling", + "expected_tools": [], + "budget": { + "max_tool_calls": 20, + "max_total_tokens": 50000 + } + } +] diff --git a/tests_ai/eval_gate.py b/tests_ai/eval_gate.py new file mode 100644 index 00000000..c7784db8 --- /dev/null +++ b/tests_ai/eval_gate.py @@ -0,0 +1,62 @@ +"""Bridge AI audit reports to offline eval budgets (obs-eval-4 live slice).""" + +from __future__ import annotations + +import json +from pathlib import Path + +from pydantic import TypeAdapter + +from tests_ai.eval_schema import EvalCase, EvalVerdict, ObservedMetrics, score_eval_case + +_EVAL_CASES = TypeAdapter(list[EvalCase]) + + +def load_eval_cases(path: Path) -> list[EvalCase]: + payload = json.loads(path.read_text(encoding="utf-8")) + return _EVAL_CASES.validate_python(payload) + + +def observed_from_report_case(case: dict[str, object]) -> ObservedMetrics: + """Read optional efficiency metrics agents may attach to a report case.""" + raw = case.get("metrics") + if not isinstance(raw, dict): + return ObservedMetrics() + tools_used = raw.get("tools_used") + tools_tuple: tuple[str, ...] = () + if isinstance(tools_used, list): + tools_tuple = tuple(str(item) for item in tools_used) + return ObservedMetrics( + tool_calls=int(raw.get("tool_calls", 0) or 0), + input_tokens=int(raw.get("input_tokens", 0) or 0), + output_tokens=int(raw.get("output_tokens", 0) or 0), + tool_errors=int(raw.get("tool_errors", 0) or 0), + step_count=int(raw.get("step_count", 0) or 0), + tools_used=tools_tuple, + ) + + +def gate_report(report: list[dict[str, object]], cases: list[EvalCase]) -> list[EvalVerdict]: + """Score report cases that include a matching ``name`` and optional ``metrics`` block.""" + cases_by_name = {case.name: case for case in cases} + verdicts: list[EvalVerdict] = [] + unknown_case_names: set[str] = set() + for entry in report: + report_cases = entry.get("cases", []) + if not isinstance(report_cases, list): + continue # malformed entry: "cases" must be a list — skip it + for case in report_cases: + if not isinstance(case, dict): + continue + name = str(case.get("name") or "") + eval_case = cases_by_name.get(name) + if eval_case is None: + # A report case with no matching eval case means the report/case + # contracts drifted — surface it instead of silently passing. + unknown_case_names.add(name) + continue + verdicts.append(score_eval_case(eval_case, observed_from_report_case(case))) + if unknown_case_names: + unknown = ", ".join(sorted(n for n in unknown_case_names if n)) + raise ValueError(f"Unknown eval case name(s) in report: {unknown}") + return verdicts diff --git a/tests_ai/eval_schema.py b/tests_ai/eval_schema.py new file mode 100644 index 00000000..4b266b22 --- /dev/null +++ b/tests_ai/eval_schema.py @@ -0,0 +1,85 @@ +"""Versioned eval-case schema + efficiency scoring (obs-eval-4, offline core). + +Shared by ``tests_ai/eval_gate.py`` and ``tests_e2e/eval_schema.py`` so AI eval +harnesses do not cross-import between sibling test packages. +""" + +from __future__ import annotations + +from typing import Literal + +from pydantic import BaseModel, Field + +EVAL_CASE_SCHEMA_VERSION: Literal[1] = 1 + + +class EfficiencyBudget(BaseModel): + """Per-scenario ceilings; ``None`` means "do not gate on this metric".""" + + max_tool_calls: int | None = Field(default=None, ge=0) + max_total_tokens: int | None = Field(default=None, ge=0) + max_tool_errors: int | None = Field(default=None, ge=0) + max_steps: int | None = Field(default=None, ge=0) + + +class EvalCase(BaseModel): + """A versioned behavioral eval scenario.""" + + # Pinned to the supported version so a mismatched payload fails closed at load. + schema_version: Literal[1] = EVAL_CASE_SCHEMA_VERSION + name: str + query: str + expected_tools: tuple[str, ...] = () + reference_outcome: str = "" + budget: EfficiencyBudget = Field(default_factory=EfficiencyBudget) + + +class ObservedMetrics(BaseModel): + """The efficiency triple observed for one scenario run.""" + + tool_calls: int = Field(default=0, ge=0) + input_tokens: int = Field(default=0, ge=0) + output_tokens: int = Field(default=0, ge=0) + tool_errors: int = Field(default=0, ge=0) + step_count: int = Field(default=0, ge=0) + tools_used: tuple[str, ...] = () + + @property + def total_tokens(self) -> int: + return self.input_tokens + self.output_tokens + + +class BudgetBreach(BaseModel): + metric: str + budget: int + observed: int + + +class EvalVerdict(BaseModel): + name: str + passed: bool + breaches: list[BudgetBreach] = Field(default_factory=list) + missing_expected_tools: tuple[str, ...] = () + + +def score_eval_case(case: EvalCase, observed: ObservedMetrics) -> EvalVerdict: + """Score a scenario: within every set budget AND used every expected tool.""" + breaches: list[BudgetBreach] = [] + + def _check(metric: str, budget: int | None, value: int) -> None: + if budget is not None and value > budget: + breaches.append(BudgetBreach(metric=metric, budget=budget, observed=value)) + + _check("tool_calls", case.budget.max_tool_calls, observed.tool_calls) + _check("total_tokens", case.budget.max_total_tokens, observed.total_tokens) + _check("tool_errors", case.budget.max_tool_errors, observed.tool_errors) + _check("step_count", case.budget.max_steps, observed.step_count) + + used = set(observed.tools_used) + missing = tuple(tool for tool in case.expected_tools if tool not in used) + return EvalVerdict( + name=case.name, + passed=not breaches and not missing, + breaches=breaches, + missing_expected_tools=missing, + ) diff --git a/tests_ai/scripts/run.py b/tests_ai/scripts/run.py index e8c4ef5d..a12acde7 100755 --- a/tests_ai/scripts/run.py +++ b/tests_ai/scripts/run.py @@ -120,6 +120,11 @@ def render_summary_line(summary: str, duration: float, *, use_color: bool, faile def main(argv: list[str] | None = None) -> int: parser = argparse.ArgumentParser() parser.add_argument("tests_dir", nargs="?", default="tests_ai") + parser.add_argument( + "--eval-cases", + default=None, + help="Optional EvalCase JSON path for post-run efficiency budget gate (obs-eval-4).", + ) args = parser.parse_args(argv) script_dir = Path(__file__).resolve().parent @@ -136,6 +141,30 @@ def main(argv: list[str] | None = None) -> int: use_color = sys.stdout.isatty() passed, failed = emit_results(report, use_color=use_color) + if args.eval_cases: + from pydantic import ValidationError + + from tests_ai.eval_gate import gate_report, load_eval_cases + + eval_cases_path = Path(args.eval_cases).resolve() + try: + cases = load_eval_cases(eval_cases_path) + budget_failures = [v for v in gate_report(report, cases) if not v.passed] + except (OSError, json.JSONDecodeError, ValidationError, ValueError) as exc: + raise SystemExit( + f"ERROR: could not evaluate --eval-cases {eval_cases_path}: {exc}" + ) from exc + for verdict in budget_failures: + print( + colorize( + f'EVAL BUDGET FAILED "{verdict.name}" breaches={verdict.breaches} ' + f"missing_tools={verdict.missing_expected_tools}", + RED, + use_color, + ) + ) + failed += len(budget_failures) + if failed: summary = f"{failed} failed" if passed: diff --git a/tests_e2e/eval_schema.py b/tests_e2e/eval_schema.py index a62f2fa9..5626cda7 100644 --- a/tests_e2e/eval_schema.py +++ b/tests_e2e/eval_schema.py @@ -1,22 +1,4 @@ -"""Versioned eval-case schema + efficiency scoring (obs-eval-4, offline core). - -Behavioral evals answer "did the task pass?" but never "did the agent take a sane, -efficient path?". A prompt or tool-description change could double the tool calls, -blow up tokens, or pick the wrong subagent while still passing the smoke reward. - -This module is the offline-testable core of obs-eval-4: - * ``EvalCase`` — a versioned scenario (query + expected tool trajectory + reference - outcome + per-scenario efficiency budgets). - * ``ObservedMetrics`` — the efficiency triple the agent loop already emits as OTel - metrics (tool calls, tokens, tool errors, step count, tools used). - * ``score_eval_case`` — compares observed vs budget and the expected trajectory. - * ``observed_from_metric_reader`` — reads the metrics back out of an in-process - OTel ``InMemoryMetricReader``, the zero-extra-plumbing tap the gap calls for. - -Deferred (the live-run slice): wiring this per-scenario into the scripted-echo e2e -suite and extending the accuracy_smoke / Harbor ``result.json`` parser — those need -a real run and a curated corpus; the schema + scorer here are their foundation. -""" +"""E2E eval schema: re-exports offline core + OTel metric reader helpers.""" from __future__ import annotations @@ -25,81 +7,27 @@ InMemoryMetricReader, NumberDataPoint, ) -from pydantic import BaseModel, Field - -EVAL_CASE_SCHEMA_VERSION = 1 - - -class EfficiencyBudget(BaseModel): - """Per-scenario ceilings; ``None`` means "do not gate on this metric".""" - - max_tool_calls: int | None = None - max_total_tokens: int | None = None - max_tool_errors: int | None = None - max_steps: int | None = None - - -class EvalCase(BaseModel): - """A versioned behavioral eval scenario.""" - - schema_version: int = EVAL_CASE_SCHEMA_VERSION - name: str - query: str - expected_tools: tuple[str, ...] = () - """Trajectory hint: tools the agent is expected to use (subset, order-agnostic).""" - reference_outcome: str = "" - budget: EfficiencyBudget = Field(default_factory=EfficiencyBudget) - - -class ObservedMetrics(BaseModel): - """The efficiency triple observed for one scenario run.""" - tool_calls: int = 0 - input_tokens: int = 0 - output_tokens: int = 0 - tool_errors: int = 0 - step_count: int = 0 - tools_used: tuple[str, ...] = () - - @property - def total_tokens(self) -> int: - return self.input_tokens + self.output_tokens - - -class BudgetBreach(BaseModel): - metric: str - budget: int - observed: int - - -class EvalVerdict(BaseModel): - name: str - passed: bool - breaches: list[BudgetBreach] = Field(default_factory=list) - missing_expected_tools: tuple[str, ...] = () - - -def score_eval_case(case: EvalCase, observed: ObservedMetrics) -> EvalVerdict: - """Score a scenario: within every set budget AND used every expected tool.""" - breaches: list[BudgetBreach] = [] - - def _check(metric: str, budget: int | None, value: int) -> None: - if budget is not None and value > budget: - breaches.append(BudgetBreach(metric=metric, budget=budget, observed=value)) - - _check("tool_calls", case.budget.max_tool_calls, observed.tool_calls) - _check("total_tokens", case.budget.max_total_tokens, observed.total_tokens) - _check("tool_errors", case.budget.max_tool_errors, observed.tool_errors) - _check("step_count", case.budget.max_steps, observed.step_count) +from tests_ai.eval_schema import ( + EVAL_CASE_SCHEMA_VERSION, + BudgetBreach, + EfficiencyBudget, + EvalCase, + EvalVerdict, + ObservedMetrics, + score_eval_case, +) - used = set(observed.tools_used) - missing = tuple(tool for tool in case.expected_tools if tool not in used) - return EvalVerdict( - name=case.name, - passed=not breaches and not missing, - breaches=breaches, - missing_expected_tools=missing, - ) +__all__ = [ + "EVAL_CASE_SCHEMA_VERSION", + "BudgetBreach", + "EfficiencyBudget", + "EvalCase", + "EvalVerdict", + "ObservedMetrics", + "observed_from_metric_reader", + "score_eval_case", +] def _counter_total(reader: InMemoryMetricReader, name: str) -> int: @@ -137,12 +65,7 @@ def _histogram_sum(reader: InMemoryMetricReader, name: str) -> int: def observed_from_metric_reader( reader: InMemoryMetricReader, *, tools_used: tuple[str, ...] = () ) -> ObservedMetrics: - """Read the efficiency triple out of an in-process OTel metric reader. - - ``tools_used`` (the trajectory) is passed in by the harness since tool names - live on metric attributes; everything else comes straight from the instruments - the agent loop already records. - """ + """Read the efficiency triple out of an in-process OTel metric reader.""" return ObservedMetrics( tool_calls=_counter_total(reader, "pythinker.tool.calls_total"), input_tokens=_counter_total(reader, "pythinker.llm.input_tokens"),