feat(core): deepen agent core — request assembly, transactional history, catalogues, toolset characterization

.coderabbit.yaml

@@ -48,7 +48,6 @@ reviews:
   path_filters:
     - "!dist/**"
     - "!build/**"
-    - "!graphify-out/**"
     - "!**/*.pyc"
     - "!**/__pycache__/**"
     # Prose and working-notes paths: keeps bot reviews focused on shipped

.gitignore

@@ -33,14 +33,9 @@ src/pythinker_code/dashboard/static/
 
 # Generated reports
 .firecrawl/
-# Graphify generated graph, cache, wiki, and Obsidian vault outputs
-graphify-out/
-graphify-out*/
 src/pythinker_code/.understand-anything/
 # understand-anything knowledge graph (any location)
 .understand-anything/
-.graphify_*.json
-.graphify_*.txt
 tests_ai/report.json
 tests_ai/terminal_bench_2_cache/
 jobs/

CHANGELOG.md

@@ -15,9 +15,48 @@ GitHub Releases page; `0.8.0` is the new starting line.
 
 ## Unreleased
 
+- **Agent-spec loading is more defensive and truthful.** Subagent `path`, `extend`, and
+  `system_prompt_path` references that resolve outside their spec's directory (or the built-in
+  agents directory) are now rejected instead of loaded, and the markdown agent catalogue no longer
+  reclassifies an unexpected parser error as a harmless "invalid field" skip — only genuinely
+  malformed frontmatter is skipped.
 - **Thinking and subagent activity now render cleanly in the terminal.** Live reasoning previews
   render complete Markdown without exposing top-level HTML comments, activity-tree rows remain
   visually stable, and the coral shimmer is reserved for the active verb spinner.
+- **Agent request compatibility is now executable and reviewable.** Provider handoff,
+  prompt ordering, persisted-versus-effective history, context JSONL restoration,
+  agent projections, and Toolset lifecycle behavior now have explicit compatibility
+  contracts guarding future agent-core changes.
+- **Skill discovery is bounded without making skills unreachable.** Pythinker now
+  searches one deterministic `SkillCatalog`, keeps exhaustive exact-name resolution,
+  and sends only task-relevant candidates to the model within an 8,000-character
+  request budget. The exhaustive `Runtime.skills` mapping remains available during
+  the compatibility window.
+- **Agent requests now have one observable assembly path.** Required guidance fails
+  closed, optional guidance reports sanitized degradation outcomes, and the new
+  `/prompt-manifest` command explains the latest request composition without storing
+  raw prompts, user text, or provenance paths.
+- **Conversation history updates are transactional.** Normal appends persist before
+  changing memory, while compaction, pruning, revert, and clear flows use atomic
+  replacement with coherent cancellation and rollback behavior. Concurrent revert
+  conflicts now stop after a bounded retry budget instead of starving indefinitely.
+  Existing JSONL records and restoration behavior remain compatible.
+- **Agent definitions now resolve through one source-aware catalogue.** YAML and
+  Markdown definitions share deterministic precedence, collision diagnostics, and
+  safe provenance handling. Unknown fields warn in this release, become errors in
+  the following minor release, and the `LaborMarket`, `AgentTypeDefinition`, and
+  generated-wrapper adapters remain through that strict-default release.
+- **Tool execution and MCP lifecycle behavior now have deterministic fault coverage.**
+  Publication rebuilds preserve the previous MCP tool registry if registration
+  fails. Characterization crossed the execution-overhead threshold, but a controlled
+  private extraction measured slightly worse and was reverted, so
+  `PythinkerToolset` remains the implementation boundary.
+- **Agent-core seams hardened from review.** Persisted usage/checkpoint records reject
+  boolean and negative token counts, `update_token_count` validates at the boundary, a
+  temporary system-prompt descriptor is closed if `fdopen` fails, request finalization
+  surfaces every provider acknowledgement failure, a failed skill projection is always
+  recorded as failed (never blurred to not-applicable), and request-assembly telemetry no
+  longer emits unbounded per-request token values as metric attributes.
 
 ## 0.57.0 (2026-07-05)
 

CLAUDE.md

@@ -9,7 +9,7 @@ Read both, in order:
 
 1. **`AGENTS.md`** — non-negotiable repository rules. Always applies.
 2. **`AGENTS.local`** — machine-specific / private local instructions (gitignored). Read it after
-   `AGENTS.md`. It may add workflow detail (e.g. the code-graph / graphify workflow) but must not
+   `AGENTS.md`. It may add workflow detail (e.g. a local code-graph workflow) but must not
    weaken or override the rules in `AGENTS.md`.
 
 @AGENTS.md

docs/.vitepress/config.ts

@@ -14,6 +14,11 @@ export default withMermaid(defineConfig({
   title: 'Pythinker Code Docs',
   description: 'Pythinker Code Documentation',
 
+  // Internal superpowers working docs (plans/specs/reports) are gitignored and
+  // never linked in the published site's nav; exclude them from VitePress's
+  // page auto-discovery so their Markdown is not compiled by the Vue SFC parser.
+  srcExclude: ['**/superpowers/**'],
+
   locales: {
     en: {
       label: 'English',

docs/en/contributing/toolset-characterization.md

@@ -0,0 +1,138 @@
+# Toolset characterization
+
+The Toolset characterization harness is a directional, local engineering aid. It uses
+deterministic no-op tools and fake MCP-style inventories to make Pythinker's framework costs
+visible without network access, credentials, hosted telemetry, or new runtime dependencies. Its
+numbers describe the machine and checkout that produced them; they are not universal product
+performance claims.
+
+Task 12 establishes the schema, evaluator, and measurement harness. It deliberately makes no
+extraction decision. Task 13 runs the full fault matrix and real five-run measurements before any
+private `_ToolExecutionPipeline`, `_McpLifecycle`, or `_ToolRegistry` extraction may be considered.
+
+## Run it
+
+From the repository root:
+
+```bash
+uv run python scripts/benchmark_toolset.py --scenario all --runs 5 \
+  --output toolset-characterization.json
+```
+
+Use `--scenario execution`, `dedupe`, `advertisement`, or `mcp` to select one family. The default is
+`all`. `--smoke` selects the smallest fixture in each chosen family and is intended only to validate
+the runner and schema:
+
+```bash
+uv run python scripts/benchmark_toolset.py --scenario all --runs 1 --smoke
+```
+
+Without `--output`, the report is written as JSON to standard output. A non-positive run count, an
+unknown scenario, or an unwritable output path exits nonzero.
+
+## Fixtures and intervals
+
+Fixture construction and warm-up are outside every named measured interval. All clocks use
+`time.monotonic_ns()`.
+
+- Execution runs parallel-safe and exclusive calls at concurrency 1, 10, and 100 through the public
+  `PythinkerToolset` facade. For `execution_mixed`, fixture `size` is a reader/writer pair count:
+  sizes 1, 10, and 100 therefore use concurrency and expected operation counts 2, 20, and 200.
+  `end_to_end` starts immediately before `handle` dispatch and ends after all returned results
+  settle. `tool_call` records absolute entry/exit intervals inside
+  deterministic no-op tools; overlapping intervals are merged before their critical-path duration
+  is subtracted from `end_to_end` for `framework_overhead`. Each call's read/write gate wait begins
+  immediately before requesting the shared or exclusive context and ends immediately after
+  admission, before `tool.call`. `read_write_gate_wait` is the union of those absolute wait
+  intervals, so overlapping queued calls count once and remain directly comparable with
+  `end_to_end` for the 25 percent gate. Every mixed scale uses an event-held reader from the first
+  pair and queues its exclusive writer before releasing it; the remaining pairs are dispatched in
+  call order behind that barrier, making contention deterministic without sleeps. The remaining
+  named lifecycle subphases are present with `measurement_status: unmeasured`
+  and an explanation because current Toolset exposes no stable boundary that would isolate them
+  without changing production behavior.
+- Dedupe dispatches two same-step calls with identical 1 KiB, 100 KiB, or 1 MiB payloads. It uses
+  the same execution intervals and keeps payload construction outside the measured region.
+- Advertisement projects 50, 500, and 5,000 real built-in, `PluginTool`, and `MCPTool` categories.
+  It records visibility policy enabled/disabled with hidden/unhidden entries, four aggregate reads
+  plus twenty individually timed repeated reads without a registry change, and a rebuild after
+  deterministic MCP publication. Each measured run records all twenty raw projection samples and
+  their nearest-rank p95; the registry repeatability decision uses the five within-run p95 values,
+  never a single projection or the 5,000-tool stress result. Setup remains
+  outside every named projection interval. Category and projection counts make the fixture behavior
+  independently checkable; the registry hash length-prefixes both policy projections in order.
+- MCP runs the current background `load_mcp_tools`/`wait_for_mcp_tools`/`cleanup` lifecycle for 1,
+  10, and 50 configured servers, replacing network connection with a deterministic local inventory
+  adapter. `mcp_lifecycle` starts immediately before background loading and ends when
+  `wait_for_mcp_tools` settles after final registry publication. `time_to_first_inventory` ends when
+  the inventory is visible through the public Toolset registry (the local all-at-once publication
+  can make first and settled effectively the same boundary), and the report records the visible
+  count at that exact publication boundary; `time_to_settled_inventory` ends after
+  `wait_for_mcp_tools` returns; and `cleanup` covers `PythinkerToolset.cleanup()` entry through
+  return. `startup_to_ready` begins immediately before the actual `Runtime.create` call and ends at
+  that same settled-inventory point.
+
+Concurrency and cancellation coordination use asyncio events and task completion, never timing
+sleeps. The harness reports cancellation completion and post-run task leakage. Its fake adapters do
+not start processes or sessions, so those leak counts remain explicit zeros. Cancellation probes
+enter real queued reader and writer states through `PythinkerToolset.handle`, cancel them, then prove
+that a later call recovers. Task 13 exercises the full failure matrix.
+
+## Repeatability and reruns
+
+Threshold decisions use exactly five isolated measured runs after warm-up. A threshold is crossed
+only when at least four of five values exceed it and the median also exceeds it. No crossings means
+uncrossed. Two or three crossings also remain uncrossed because the repeatability rule did not pass.
+A single crossing is treated as an outlier and therefore inconclusive; it requests one complete
+five-run rerun. The rerun replaces the inconclusive set for the decision, and a second rerun is never
+requested.
+
+The pure evaluator records both primary and rerun values, the selected crossing count and median,
+the final `crossed`, `uncrossed`, or `inconclusive` state, and whether a rerun is still required.
+
+## Extraction thresholds
+
+Task 13 applies these approved gates with the repeatability rule:
+
+- Execution pipeline: non-tool framework overhead exceeds 10 percent of p95 latency for short
+  in-process tools, or three independent recent changes repeatedly touch the same lifecycle region
+  and extraction deletes that shared state.
+- MCP lifecycle: the defined lifecycle interval exceeds 20 percent of startup-to-ready with ten
+  servers; cleanup exceeds six seconds; a task, process, session, or publication leak is reproduced;
+  or three modules directly require MCP lifecycle state.
+- Registry: advertisement exceeds 5 ms p95 at 500 tools, or recurring collision/rebuild/visibility
+  defects would be eliminated by one owner. The 5,000-tool stress fixture cannot trigger extraction
+  by itself.
+- Read/write gate: gate wait exceeds 25 percent of end-to-end p95 in a realistic mixed workload, or
+  a second real consumer appears.
+
+No threshold crossing, file length alone, duplicate old/new state, a new public interface, a
+callback cycle, or a red characterization/cancellation test mandates no extraction.
+
+## JSON schema
+
+The versioned report contains:
+
+- `environment`: Python version and implementation plus platform information;
+- `scenarios[].fixture`: scenario kind, size, concurrency, payload size, and composition; mixed
+  execution explicitly reports `composition: reader/writer pairs`;
+- `warmups` and `iterations`;
+- `phases`: measured/unmeasured status, raw nanosecond samples, optional raw within-run sample
+  groups, median, nearest-rank p95, throughput where meaningful, and an explanation for unavailable
+  boundaries;
+- deterministic `registry_hash`;
+- `allocation_peak_bytes` and `retained_object_delta` from `tracemalloc`;
+- `cancellation`: completion status and completion duration;
+- `leaks`: pending task, process, and session counts;
+- `task_count_peak`: the maximum observed asyncio task count during a measured sample;
+- `operation_count`, `category_counts`, and `projection_counts`: independently checkable fixture
+  execution and advertisement behavior;
+- `lifecycle_status`: `completed` for local execution/registry fixtures or `settled` after the fake
+  MCP inventory is fully published; and
+- `decisions`: threshold, five primary values, optional five-value rerun, crossing count, median,
+  primary/rerun/final states, and rerun-required state.
+
+Partial, smoke, and non-five-run reports keep an empty `decisions` array. The documented full
+`--scenario all --runs 5` command deterministically derives all seven Task 13 decisions from the raw
+scenario samples and writes them in the same report. A human-readable crossed/uncrossed/inconclusive
+decision record remains a Task 13 deliverable after deterministic fault tests are complete.

docs/en/customization/agents.md

@@ -59,6 +59,21 @@ Recognized frontmatter fields are `name`, `description`, `tools`, `model`, and
 warning. Discovered markdown agents appear as `Agent` tool subagent types alongside the
 built-in types.
 
+### Agent definition validation rollout
+
+Pythinker currently resolves YAML and repository markdown definitions into one agent catalogue.
+Unknown definition fields are accepted with one aggregated startup warning per source in this
+release. Warnings identify the field path but do not include field values, prompt content, or raw
+absolute source paths.
+
+The production policy will reject unknown fields in the immediately following minor release.
+Fix warnings before upgrading: required YAML definitions will then fail to load, while an invalid
+optional markdown definition will be skipped with a diagnostic. The existing `LaborMarket`
+compatibility interface and generated markdown YAML wrappers will remain for that strict-default
+release. Their earliest removal is one additional minor release later, and only after direct
+catalogue launch has equivalent prompt, model, tool-policy, required-MCP, foreground, and
+background behavior.
+
 ## Custom agent files
 
 Agents are defined in YAML format. Load a custom agent with the `--agent-file` flag:

docs/en/reference/slash-commands.md

@@ -38,6 +38,13 @@ Aliases: `/report`
 
 List the available subagent types, showing each agent's name, when to use it, its default model, and its tool posture.
 
+### `/prompt-manifest`
+
+Show the latest request-assembly status for the current session. The output includes opaque,
+stable fragment identifiers, admission outcomes, and token estimates. It never includes prompt
+content, user content, raw source names, raw file paths, credentials, or stack traces. Before the
+first assembled request, the command reports that no manifest is available.
+
 ## Account and configuration
 
 ### `/login`

docs/en/release-notes/changelog.md

@@ -17,9 +17,48 @@ GitHub Releases page; `0.8.0` is the new starting line.
 
 ## Unreleased
 
+- **Agent-spec loading is more defensive and truthful.** Subagent `path`, `extend`, and
+  `system_prompt_path` references that resolve outside their spec's directory (or the built-in
+  agents directory) are now rejected instead of loaded, and the markdown agent catalogue no longer
+  reclassifies an unexpected parser error as a harmless "invalid field" skip — only genuinely
+  malformed frontmatter is skipped.
 - **Thinking and subagent activity now render cleanly in the terminal.** Live reasoning previews
   render complete Markdown without exposing top-level HTML comments, activity-tree rows remain
   visually stable, and the coral shimmer is reserved for the active verb spinner.
+- **Agent request compatibility is now executable and reviewable.** Provider handoff,
+  prompt ordering, persisted-versus-effective history, context JSONL restoration,
+  agent projections, and Toolset lifecycle behavior now have explicit compatibility
+  contracts guarding future agent-core changes.
+- **Skill discovery is bounded without making skills unreachable.** Pythinker now
+  searches one deterministic `SkillCatalog`, keeps exhaustive exact-name resolution,
+  and sends only task-relevant candidates to the model within an 8,000-character
+  request budget. The exhaustive `Runtime.skills` mapping remains available during
+  the compatibility window.
+- **Agent requests now have one observable assembly path.** Required guidance fails
+  closed, optional guidance reports sanitized degradation outcomes, and the new
+  `/prompt-manifest` command explains the latest request composition without storing
+  raw prompts, user text, or provenance paths.
+- **Conversation history updates are transactional.** Normal appends persist before
+  changing memory, while compaction, pruning, revert, and clear flows use atomic
+  replacement with coherent cancellation and rollback behavior. Concurrent revert
+  conflicts now stop after a bounded retry budget instead of starving indefinitely.
+  Existing JSONL records and restoration behavior remain compatible.
+- **Agent definitions now resolve through one source-aware catalogue.** YAML and
+  Markdown definitions share deterministic precedence, collision diagnostics, and
+  safe provenance handling. Unknown fields warn in this release, become errors in
+  the following minor release, and the `LaborMarket`, `AgentTypeDefinition`, and
+  generated-wrapper adapters remain through that strict-default release.
+- **Tool execution and MCP lifecycle behavior now have deterministic fault coverage.**
+  Publication rebuilds preserve the previous MCP tool registry if registration
+  fails. Characterization crossed the execution-overhead threshold, but a controlled
+  private extraction measured slightly worse and was reverted, so
+  `PythinkerToolset` remains the implementation boundary.
+- **Agent-core seams hardened from review.** Persisted usage/checkpoint records reject
+  boolean and negative token counts, `update_token_count` validates at the boundary, a
+  temporary system-prompt descriptor is closed if `fdopen` fails, request finalization
+  surfaces every provider acknowledgement failure, a failed skill projection is always
+  recorded as failed (never blurred to not-applicable), and request-assembly telemetry no
+  longer emits unbounded per-request token values as metric attributes.
 
 ## 0.57.0 (2026-07-05)