feat(tui+lsp): TUI theme system, streaming phase 0, LSP subsystem, and report panel polish

AGENTS.md

@@ -224,6 +224,41 @@ Pick the smallest reliable gate for the change, then run broader gates before re
 If a gate cannot run because of missing system tools (for example `npm`), report that explicitly
 instead of claiming success.
 
+## Pre-PR gate (run before pushing or opening a PR)
+
+CI failures that "slip to GitHub" almost always trace to pushing after a *partial* local check
+(for example running `ruff`/`pyright` on a single file instead of the whole package). Before you
+push a branch or open a PR that touches shipped code, run the full gate and clear every item below.
+Running a focused check on only the files you edited is **not** sufficient — snapshot, static, and
+bundling tests fail on files you did not touch.
+
+1. **Full gate, not partial.** Run `make check-pythinker-code` (ruff + format + pyright) **and**
+   `make test-pythinker-code`. Paste/confirm the actual "All checks passed" / passing summary — a
+   green `ruff` alone is not a green `check` (pyright and format are separate). For changes to
+   another workspace package, run that package's `make check-* && make test-*` too.
+2. **Include `tests_e2e`.** CI runs `tests` and `tests_e2e`. New slash commands, wire events, or
+   agent-spec/tool changes move the wire-handshake snapshot and the agent-spec/config/pyinstaller
+   snapshots. Re-run the affected tests; apply deliberate snapshot updates with
+   `uv run pytest <node> --inline-snapshot=fix` and **read the resulting diff** before committing.
+3. **Snapshot fix-direction.** A hardcoded expected value that changed deliberately → update the
+   **test**. An invariant of the form "two values must stay equal" (e.g. a prompt token that must
+   track a core theme token) → fix the **code** that drifted, never the test.
+4. **New source files clear the static checks.** `tests/test_ai_static_requirements.py` enforces
+   explicit text encoding (`encoding="utf-8"`, `errors="replace"` for tool decodes) and other
+   invariants across `src/pythinker_code/**`. Note the ruff-vs-static conflict: ruff `UP012` strips
+   `"utf-8"` from a **string-literal** `.encode("utf-8")`, but the static check wants an explicit
+   `encoding=`. Encode/decode via a **local variable or call result**, not a literal, so both gates
+   pass (see `lsp/framing.py`).
+5. **New bundled files/tools update the manifests.** A new tool `*.md`, prompt, or package adds
+   entries to `tests/utils/test_pyinstaller_utils.py` (`datas` + `hiddenimports`) and, for new
+   config keys, `tests/core/test_config.py::test_default_config_dump`.
+6. **Changelog.** Any change to shipped paths (`src/*`, `packages/*`, installers, release
+   workflows, `pythinker.spec`) needs a new `- ...` line under `## Unreleased` in `CHANGELOG.md`, or
+   the `changelog-entry-required` check fails the PR.
+7. **Confirm what is actually new.** Diff against `origin/main` (`git log origin/main..HEAD`,
+   `git diff origin/main...HEAD --stat`) so the PR scope — and the review/verification surface — is
+   what you intend, not stale local commits.
+
 ## Project architecture
 
 ### Runtime path

CHANGELOG.md

@@ -15,6 +15,87 @@ GitHub Releases page; `0.8.0` is the new starting line.
 
 ## Unreleased
 
+- **TUI composing preview wraps space-aligned report prose cleanly.** The
+  streaming preview now runs the same lightweight space-column normalizer used at
+  finalize and wraps long `Severity`/`Location`/`What` rows with a hanging
+  continuation indent, so wrapped fragments no longer orphan at column 0.
+- **ToolSearch hidden from models that can't use it.** `ToolSearch` is now offered
+  only when the active model genuinely supports the deferred tool-search workflow
+  (Anthropic's `tool_reference`/`defer_loading` beta on `api.anthropic.com`). The
+  `type="anthropic"` compat proxies (z.ai/GLM, Kimi, MiniMax, opencode) and all
+  non-Anthropic providers no longer see it, fixing a loop where weaker tool-callers
+  (e.g. GLM-5.2) repeatedly "searched" for tools instead of calling them. Override
+  with `ENABLE_TOOL_SEARCH=true|false`. The tool's description no longer claims that
+  hidden/deferred tools exist (pythinker loads no tools lazily), removing the prompt
+  that primed the loop in the first place.
+- **Output-token-limit nudge text aligned with reference.** The system-reminder injected when a response is cut off by the output token limit now matches the reference byte-exactly: "Output token limit hit. Resume directly — no apology, no recap of what you were doing. Pick up mid-thought if that is where the cut happened. Break remaining work into smaller pieces."
+- **`SetTodoList` accepts Cursor-style todo payloads.** Todo items sent with `content` instead of `title` (the shape models learn from Cursor/Claude `TodoWrite`) are normalized at the validation boundary (`content` → `title` when `title` is absent; canonical `title` wins; `content` is dropped) and persist as title-only session state instead of failing with missing-`title` errors.
+- **Failed `SetTodoList` cards stay compact.** Validation failures no longer render a broken todo tree with blank labels plus a raw Pydantic dump; the card shows a short actionable summary (with full detail only when expanded).
+- **ToolSearch scrollback suppression.** Consecutive `ToolSearch` probes during deferred tool discovery are now collapsed: only the last probe in each run is shown in the transcript, mirroring the blackbox `isAbsorbedSilently` contract. Intermediate discovery calls no longer produce repeated "Tools(…)" lines.
+- **Bare skill/flow slash names.** The slash menu now matches `skill:`/`flow:`
+  commands on their bare segment, so typing `/designer` (or `/design`) surfaces
+  `/skill:designer-skill`; accepting inserts the canonical command name. When no
+  prefix matches, a fuzzy fallback surfaces the distinctive word even when
+  misspelled (`/gurd` → `/skill:pythinker-guard`), so skills sharing a common
+  prefix stay reachable.
+- **TUI composing preview gap.** Removed the visible double-blank row between
+  `Composing…` and the in-progress preview (leading newline from commit
+  boundaries no longer leaks through the plain-text preview path), and aligned
+  the Rich `Live` paint rate with the 25 Hz reveal scheduler (was 10 Hz).
+- **ToolSearch TUI display.** `ToolSearch` results now render as a compact
+  "N tools discovered (Agent, Grep, …)" summary instead of dumping the full
+  tool catalog with descriptions; ctrl+o expands to tool names only.
+- **Tool header highlights.** Read/Write/Edit/Grep and similar tool-call subjects
+  now use the brand periwinkle `accent` token instead of cyan `info`; line ranges
+  stay on the yellow `warning` token.
+- **pythinker-x theme port.** Diff palette, 32 bundled syntax theme names, Catppuccin
+  Frappe/Macchiato styles, and `/theme code` syntax picker aligned with the Pythinker-X TUI.
+- **TUI inline code color.** Inline `` `code` `` highlights and the `pythinker-ansi`
+  syntax theme now use brand periwinkle/accent and blue ANSI roles instead of cyan.
+- **TUI transcript spacing.** User prompts leave one blank row before the agent stream
+  starts; finished tool cards and flushed agent paragraphs leave a trailing blank row
+  before the next block (Bash/Read output → next ⏺ paragraph, etc.).
+- **Welcome banner colors.** Branch uses light neutral grey; model name uses the muted
+  yellow warning token.
+- **TUI theme package.** Centralize dark/light palettes, prompt classes, and Rich/PTK
+  adapters in `ui/theme/` with `/theme current|doctor|tokens` inspection commands.
+- **TUI diff markers.** Inline diff rows now leave a space after `+`/`-` markers so
+  `@`-prefixed lines (e.g. CSS `@keyframes`) do not run together with the sign.
+- **Composing block spacing.** Staged agent paragraphs keep one blank row before the
+  Composing activity line while the stream is still live.
+- **Slash input UX.** Prefix-highlight skills and plugins while typing; ghost-complete
+  and highlight fixed subcommands such as `/theme current`.
+- **TUI streaming smoothness (Phase 0).** Coalesce Rich Live repaints to a 25 Hz frame budget,
+  render live previews as plain text (no per-token markdown re-parse), stage committed slices
+  inside the Live region until finalize, and use a fixed-width blinking streaming caret that
+  does not reflow wrapped lines.
+- **LSP code intelligence.** Plugin-provided language servers power a new `LSP` agent tool
+  (go-to-definition, find-references, hover, symbols, call hierarchy) with session-scoped
+  server lifecycle, passive diagnostics injected after file edits, and plugin-based server
+  discovery/recommendation — no bundled language-server binaries.
+- **Token activity card.** `/usage daily|weekly|cumulative` (and the bare `/usage` default
+  when no provider adapter is configured) now render a 52-week × 7-day heatmap of
+  total tokens consumed each day, with a `Lifetime · Peak · Streak · Longest task` summary
+  line and a footer that lets the user switch between daily/weekly/cumulative views. Data is
+  read from the local session wire files; the per-provider adapter behavior is unchanged.
+- **RunAgents tolerates blank list entries.** Models occasionally emit bare `"\n"` strings
+  between the agent objects in the `agents` array; those are now stripped before validation so
+  a multi-agent launch no longer fails with a validation error, while genuinely invalid entries
+  are still rejected.
+- **Report panel rendering.** Standardized report panels render only the panel title and section
+  headers bold (body prose stays regular weight), tag finding locations with a file marker, and
+  use a dedicated `secondary` theme token for scope/note text.
+- **Theme token consistency.** The dark prompt frame/separator/dialog borders and the prompt
+  glyph now track their canonical core theme tokens, and inline code spans correctly drop an
+  inherited background.
+- **External approvals repaint promptly.** Out-of-band approval requests and steer input now
+  force an immediate live-view repaint instead of waiting for the streaming frame budget, and
+  the live-view refresh loop is supervised so a refresh-loop failure surfaces instead of
+  silently freezing the view.
+- **LSP robustness.** Bounded JSON-RPC frame size and graceful-shutdown timeout, document
+  version tracking for `didChange`, open-document state cleared on server restart, empty
+  diagnostics payloads clear stale entries, and tightened `/usage` activity-argument validation.
+
 ## 0.47.0 (2026-06-16)
 
 - **Plugin marketplaces and activation policy.** `pythinker plugin marketplace` can add,

docs/.vitepress/config.ts

@@ -48,6 +48,7 @@ export default withMermaid(defineConfig({
               text: 'Customization',
               items: [
                 { text: 'Model Context Protocol', link: '/en/customization/mcp' },
+                { text: 'Language Server Protocol', link: '/en/customization/lsp' },
                 { text: 'Plugins (Beta)', link: '/en/customization/plugins' },
                 { text: 'Hooks (Beta)', link: '/en/customization/hooks' },
                 { text: 'Agent Skills', link: '/en/customization/skills' },

docs/en/customization/architecture.md

@@ -100,7 +100,7 @@ canonical list lives in `src/pythinker_code/wire/types.py` (`Event` union): `Ste
 | `src/pythinker_code/soul/context.py` | Conversation history, checkpoints, JSONL persistence. | `Context` |
 | `src/pythinker_code/soul/toolset.py` | Loads built-in + MCP tools, injects deps, executes calls. | `PythinkerToolset` |
 | `src/pythinker_code/soul/slash.py` | Slash-command registry and dispatch. | `registry` |
-| `src/pythinker_code/soul/dynamic_injection.py` (+ `dynamic_injections/`) | Injects budgeted `<system-reminder>` content per step: plan-mode, auto-mode, model-defense. | `DynamicInjectionProvider` |
+| `src/pythinker_code/soul/dynamic_injection.py` (+ `dynamic_injections/`) | Injects budgeted `<system-reminder>` content per step: plan-mode, auto-mode, model-defense, LSP diagnostics. | `DynamicInjectionProvider` |
 | `src/pythinker_code/soul/permission.py` | Per-step permission profiles (`read_only`/`plan`/`ask`/`implement`/`review`/`verify`) and destructiveness classification. | `tool_destructive_reason`, `shell_command_signature` |
 | `src/pythinker_code/soul/denwarenji.py` | D-Mail checkpoint rewind (`BackToTheFuture`). | — |
 | `src/pythinker_code/soul/flow_runner.py` | Ralph Loop driver for `/flow` and iterative commands. | — |
@@ -142,6 +142,7 @@ wrapped with `UntrustedData`.
 | `src/pythinker_code/tools/file/` | `ReadFile`, `WriteFile`, `StrReplaceFile`, `Glob`, `Grep`, `ReadMediaFile` |
 | `src/pythinker_code/tools/shell/` | `Shell` |
 | `src/pythinker_code/tools/web/` | `SearchWeb`, `FetchURL` (conditional on deps) |
+| `src/pythinker_code/tools/lsp/` | `Lsp` (model name `LSP`; plugin-backed language servers) |
 | `src/pythinker_code/tools/agent/` | `Agent`, `RunAgents` |
 | `src/pythinker_code/tools/background/` | `TaskOutput`, `TaskList`, `TaskInput`, `TaskStop`, `TaskHandoff` |
 | `src/pythinker_code/tools/` (other) | `AskUserQuestion`, `EnterPlanMode`/`ExitPlanMode`, `Think`, `SetTodoList`, `Memory`, `Recall`, `Scratchpad`, `Suggest`, `Progress`, `ReadSkill`, `SendDMail`, `ListMcpResources`/`ReadMcpResource` |
@@ -201,7 +202,19 @@ Full session lifecycle: `initialize`, `new_session`, `load_session`, `resume_ses
 | --- | --- | --- |
 | `src/pythinker_code/skill/`, `src/pythinker_code/skills/` | Skill discovery/loading across scopes (project > user > extra > built-in), local specialization, flow skills; injected via `PYTHINKER_SKILLS`. Bundled skills live in `skills/`. | `Skill`, `discover_skills_from_roots`, `index_skills`, `format_skills_for_prompt`, `Flow`, `SkillLockFile` |
 | `src/pythinker_code/hooks/` | Lifecycle hook engine: 13 events, server-side shell commands and client-side Wire subscriptions; fail-open (block only on explicit exit code 2 / structured deny). | `HookEngine`, `HookDef`, `HookEventType`, `HOOK_EVENT_TYPES`, `run_hook`, `events` |
-| `src/pythinker_code/plugin/` | Plugin discovery, install (local/git/zip with SSRF + traversal guards, staged atomic install), and subprocess tool execution with fresh credential injection. | `parse_plugin_json`, `PluginSpec`, `install_plugin`, `list_plugins`, `load_plugin_tools`, `PluginTool` |
+| `src/pythinker_code/plugin/` | Plugin discovery, install (local/git/zip with SSRF + traversal guards, staged atomic install), subprocess tool execution, MCP and LSP server configs (`plugin_lsp_servers`). | `parse_plugin_json`, `PluginSpec`, `install_plugin`, `list_plugins`, `load_plugin_tools`, `PluginTool`, `plugin_mcp_servers`, `plugin_lsp_servers` |
+
+## LSP subsystem
+
+Session-scoped language-server processes over `Host.exec` stdio (JSON-RPC Content-Length framing).
+Servers are plugin-only; one `LspService` per `Runtime`, shared by subagents, torn down in
+`cleanup_runtime_resources()`.
+
+| Path | Purpose | Key entry points |
+| --- | --- | --- |
+| `src/pythinker_code/lsp/` | Client, server lifecycle, routing, diagnostics registry, plugin loader, recommendation. | `LspService`, `LspServerManager`, `LspClient`, `DiagnosticRegistry`, `plugin_lsp_servers` |
+
+Trust boundary: LSP subprocesses run with agent privileges; output is untrusted project content.
 
 ## Memory, background, and notifications
 

docs/en/customization/lsp.md

@@ -0,0 +1,80 @@
+# Language Server Protocol (LSP)
+
+Pythinker Code can connect to [Language Server Protocol](https://microsoft.github.io/language-server-protocol/) servers for semantic code intelligence: go-to-definition, find-references, hover, symbols, and call hierarchy.
+
+## Plugin-only servers
+
+LSP servers are **not** configured in user or project TOML. They come only from installed plugins — inline `lspServers` in `plugin.json` or a plugin-root `.lsp.json` file. Pythinker does not bundle language-server binaries.
+
+Enable executable plugin artifacts (`plugins.external_exec = true` or `pythinker plugin enable <name>`) so plugin-provided LSP subprocesses are allowed.
+
+### Server config schema
+
+Each entry in `lspServers` (or the root object of `.lsp.json`) maps a server name to a config object:
+
+```json
+{
+  "my-server": {
+    "command": "pylsp",
+    "args": ["--check-parent-process"],
+    "extensionToLanguage": { ".py": "python" },
+    "env": { "VIRTUAL_ENV": "${VIRTUAL_ENV:-}" },
+    "initializationOptions": {},
+    "startupTimeout": 30.0,
+    "maxRestarts": 3
+  }
+}
+```
+
+| Field | Required | Description |
+|-------|----------|-------------|
+| `command` | yes | Executable to launch |
+| `args` | no | Additional CLI arguments |
+| `extensionToLanguage` | yes | Maps file extensions to LSP language IDs |
+| `env` | no | Extra environment variables for the server process |
+| `initializationOptions` | no | Passed verbatim in the LSP `initialize` request |
+| `startupTimeout` | no | Seconds to wait for server ready (default `30.0`, must be `> 0`) |
+| `maxRestarts` | no | Max automatic restarts on crash (default `3`, `0` disables) |
+
+Values in `command`, `args`, and `env` support `${VAR}` and `${VAR:-default}` expansion against the process environment. Two plugin-local path variables are always available: `${PYTHINKER_PLUGIN_ROOT}` (the plugin directory) and `${PYTHINKER_PLUGIN_DATA}` (a writable per-plugin data directory). The `CLAUDE_PLUGIN_ROOT` / `CLAUDE_PLUGIN_DATA` spellings are accepted as aliases.
+
+## Agent tool
+
+The `LSP` tool is available on the default agent and the `coder` subagent (not on read-only profiles such as `code_reviewer`). It exposes nine operations with 1-based line/character positions (editor-style).
+
+| Operation | LSP method |
+|-----------|------------|
+| `goToDefinition` | `textDocument/definition` |
+| `findReferences` | `textDocument/references` |
+| `hover` | `textDocument/hover` |
+| `documentSymbol` | `textDocument/documentSymbol` |
+| `workspaceSymbol` | `workspace/symbol` |
+| `goToImplementation` | `textDocument/implementation` |
+| `prepareCallHierarchy` | `textDocument/prepareCallHierarchy` |
+| `incomingCalls` | `callHierarchy/incomingCalls` |
+| `outgoingCalls` | `callHierarchy/outgoingCalls` |
+
+Results from `findReferences`, `goToDefinition`, `goToImplementation`, and `workspaceSymbol` automatically filter out paths that match the project's `.gitignore`.
+
+Servers start lazily on first use per language and stay alive for the session. Subagents share the root session's LSP processes.
+
+## Passive diagnostics
+
+After `WriteFile` or `StrReplaceFile`, the session notifies open language servers and surfaces new compiler/linter diagnostics on the next turn via dynamic context injection (budget-capped). Diagnostics are labeled as LSP-reported, not agent-asserted.
+
+## Configuration
+
+Feature switches only — in `~/.pythinker/config.toml`:
+
+```toml
+[lsp]
+enabled = true
+recommendation_disabled = false
+recommendation_never = []
+```
+
+When you edit a file whose extension matches a discoverable but not-yet-installed plugin server, Pythinker may suggest installing that plugin (respecting `recommendation_never` and auto-disabling after repeated ignores).
+
+## Trust boundary
+
+LSP servers run as subprocesses with the agent's privileges. Hover, symbol, and diagnostic text is treated as untrusted project content (same class as `ReadFile` output).