From 1d12ac168e998a7b43a5b7a6ad7dcdf054c68082 Mon Sep 17 00:00:00 2001 From: M Waleed Kadous Date: Tue, 28 Jul 2026 05:19:04 -0700 Subject: [PATCH] [Spec 1252] Harvest: 17-file drift reconciliation + knowledge artifacts The harvest of Spec 1252 after the pr-gate decision not to merge the full machinery (PR #1278, closed unmerged; charter-level re-scope -- a successor shrinks the surface radically before enforcement is rebuilt fit-for-purpose). RECONCILIATION (the live bug): all 17 drifted codev/ shadow copies synced to the skeleton content they had silently fallen behind. Served SPIR/ASPIR builder prompts regain their Multi-PR Mechanics and Verify Phase sections and the 'Entering verify phase.' notification -- until now, adopters received verify instructions and this repo's builders did not. Architect rulings applied: E1 max_iterations: 3 for spir/aspir (decided on the measured evidence that review rounds run 1-2, mean 1.12); E2 the air/bugfix check cwds live in .codev/config.json porch.checks (Spec #550 mechanism), so the forked protocol.json files sync to the skeleton. Spec-746 pure-addition baselines re-derived for the two synced builder-prompts (frozen-file baselines break on any legitimate edit). KNOWLEDGE ARTIFACTS: the 77-row shadow-tree audit with architect rulings and a harvest note; word-count baseline and after-measurement (the honest -7.0% N1 record with its decomposition); the behavioural baseline (B1 51.88% REQUEST_CHANGES over 160 verdicts, self-excluded, B5 snapshot); the measurement tooling (surface script, behaviour metrics library + CLI + determinism test); the project review telling the true story including the non-merge decision; and the builder thread. The scar-registry concept and the eight ratified rule wordings move into the successor charter -- deferred, not discarded. Full suite from a normal checkout path: green (the three worktree-write-guard failures observed in the tmp-located build worktree are the documented /private/tmp-allowlist platform artifact; the suite is untouched by this change). --- codev/protocols/air/protocol.json | 3 - codev/protocols/air/protocol.md | 1 - codev/protocols/aspir/builder-prompt.md | 24 +- codev/protocols/aspir/protocol.json | 10 +- codev/protocols/aspir/templates/plan.md | 20 - codev/protocols/aspir/templates/spec.md | 28 - codev/protocols/bugfix/builder-prompt.md | 2 +- codev/protocols/bugfix/protocol.json | 2 - codev/protocols/experiment/protocol.md | 7 +- codev/protocols/protocol-schema.json | 2 +- codev/protocols/spike/protocol.md | 2 +- codev/protocols/spir/builder-prompt.md | 25 +- codev/protocols/spir/protocol.json | 10 +- codev/protocols/spir/protocol.md | 138 +-- codev/protocols/spir/templates/plan.md | 20 - codev/protocols/spir/templates/spec.md | 28 - codev/resources/1252-behavior-baseline.md | 166 ++++ codev/resources/1252-shadow-tree-audit.md | 282 ++++++ codev/resources/1252-word-after-phase7.md | 58 ++ codev/resources/1252-word-baseline.md | 39 + .../1252-prompt-architecture-single-own.md | 152 ++++ codev/roles/architect.md | 29 +- codev/state/spir-1252_thread.md | 826 ++++++++++++++++++ .../codev/scripts/measure-prompt-behavior.ts | 19 + .../__tests__/prompt-behavior-metrics.test.ts | 187 ++++ .../aspir-builder-prompt.md.baseline | 40 +- .../baselines/spir-builder-prompt.md.baseline | 41 +- .../codev/src/lib/prompt-behavior-metrics.ts | 388 ++++++++ scripts/measure-prompt-surface.sh | 140 +++ 29 files changed, 2448 insertions(+), 241 deletions(-) create mode 100644 codev/resources/1252-behavior-baseline.md create mode 100644 codev/resources/1252-shadow-tree-audit.md create mode 100644 codev/resources/1252-word-after-phase7.md create mode 100644 codev/resources/1252-word-baseline.md create mode 100644 codev/reviews/1252-prompt-architecture-single-own.md create mode 100644 codev/state/spir-1252_thread.md create mode 100644 packages/codev/scripts/measure-prompt-behavior.ts create mode 100644 packages/codev/src/__tests__/prompt-behavior-metrics.test.ts create mode 100644 packages/codev/src/lib/prompt-behavior-metrics.ts create mode 100755 scripts/measure-prompt-surface.sh diff --git a/codev/protocols/air/protocol.json b/codev/protocols/air/protocol.json index 7fea5bec9..b634eae26 100644 --- a/codev/protocols/air/protocol.json +++ b/codev/protocols/air/protocol.json @@ -30,13 +30,11 @@ "checks": { "build": { "command": "npm run build", - "cwd": "packages/codev", "on_fail": "retry", "max_retries": 2 }, "tests": { "command": "npm test -- --exclude='**/e2e/**'", - "cwd": "packages/codev", "description": "Unit tests only - e2e tests run in PR phase", "on_fail": "retry", "max_retries": 2 @@ -72,7 +70,6 @@ }, "e2e_tests": { "command": "npm run test:e2e 2>&1 || echo 'e2e tests skipped (not configured)'", - "cwd": "packages/codev", "description": "Full e2e test suite", "optional": true } diff --git a/codev/protocols/air/protocol.md b/codev/protocols/air/protocol.md index 8552d3c3b..74609fd29 100644 --- a/codev/protocols/air/protocol.md +++ b/codev/protocols/air/protocol.md @@ -34,7 +34,6 @@ AIR is a minimal protocol for implementing small features (< 300 LOC) where the - Bug fixes → use **BUGFIX** - Features needing spec discussion → use **SPIR** or **ASPIR** -- Amendments to existing specs → use **TICK** - Architectural changes → use **SPIR** - Complex features with multiple phases → use **SPIR** or **ASPIR** diff --git a/codev/protocols/aspir/builder-prompt.md b/codev/protocols/aspir/builder-prompt.md index 07d02b701..d303da59e 100644 --- a/codev/protocols/aspir/builder-prompt.md +++ b/codev/protocols/aspir/builder-prompt.md @@ -68,11 +68,33 @@ By default, the PR is opened during/after the final implement phase, with all ph The architect MAY request a PR at any point — for spec review, mid-implementation feedback, slicing a large spec into shippable PRs, etc. When the architect explicitly asks for a PR earlier (or for additional PRs), follow that direction. The prohibition is specifically on the *builder* autonomously deciding to open per-phase PRs without architect request. +### Multi-PR Mechanics (when the architect requests sequential PRs) + +Your worktree is persistent — it survives across PR merges. When the architect asks for sequential PRs (e.g., to slice a large spec into shippable pieces), use this loop: + +1. Cut a branch, open a PR, wait for merge +2. After merge: `git fetch origin && git checkout -b origin/` — where `` is the branch the architect targets PRs at (usually `main`; check the open PR's `baseRefName` if unsure) +3. Continue to the next slice, open another PR + +**Important**: Do NOT run `git checkout ` — git worktrees cannot check out a branch that's checked out elsewhere. Always branch off `origin/` via fetch. + +Record PRs: `porch done {{project_id}} --pr --branch ` +Record merges: `porch done {{project_id}} --merged ` + +## Verify Phase + +After the final PR merges, the project enters the **verify** phase. You stay alive through verify: +1. Pull the integration branch into your worktree +2. Run `porch done {{project_id}}` to signal verification is ready +3. The architect approves `verify-approval` when satisfied + +If verification is not needed: `porch verify {{project_id}} --skip "reason"` + ## Notifications Always use `afx send architect "..."` to notify the architect at key moments: - **Gate reached**: `afx send architect "Project {{project_id}}: ready for approval"` - **PR ready**: `afx send architect "PR #N ready for review (project {{project_id}})"` -- **PR merged**: `afx send architect "Project {{project_id}} complete. PR merged. Ready for cleanup."` +- **PR merged**: `afx send architect "Project {{project_id}} PR merged. Entering verify phase."` - **Blocked**: `afx send architect "Blocked on project {{project_id}}: [reason]"` ## Handling Flaky Tests diff --git a/codev/protocols/aspir/protocol.json b/codev/protocols/aspir/protocol.json index ed0dd7edc..fb5c4f993 100644 --- a/codev/protocols/aspir/protocol.json +++ b/codev/protocols/aspir/protocol.json @@ -23,7 +23,7 @@ "models": ["gemini", "codex", "claude"], "parallel": true }, - "max_iterations": 8, + "max_iterations": 3, "on_complete": { "commit": true, "push": true @@ -44,7 +44,7 @@ "models": ["gemini", "codex", "claude"], "parallel": true }, - "max_iterations": 8, + "max_iterations": 3, "on_complete": { "commit": true, "push": true @@ -70,7 +70,7 @@ "models": ["gemini", "codex", "claude"], "parallel": true }, - "max_iterations": 8, + "max_iterations": 3, "on_complete": { "commit": true, "push": true @@ -107,7 +107,7 @@ "models": ["gemini", "codex", "claude"], "parallel": true }, - "max_iterations": 8, + "max_iterations": 3, "on_complete": { "commit": true, "push": true @@ -160,7 +160,7 @@ }, "defaults": { "mode": "strict", - "max_iterations": 8, + "max_iterations": 3, "verify": { "models": ["gemini", "codex", "claude"], "parallel": true diff --git a/codev/protocols/aspir/templates/plan.md b/codev/protocols/aspir/templates/plan.md index 831199848..9da106498 100644 --- a/codev/protocols/aspir/templates/plan.md +++ b/codev/protocols/aspir/templates/plan.md @@ -182,23 +182,3 @@ Phase 1 ──→ Phase 2 ──→ Phase 3 ## Notes [Additional context, assumptions, or considerations] ---- - -## Amendment History - -This section tracks all TICK amendments to this plan. TICKs modify both the spec and plan together as an atomic unit. - - - - \ No newline at end of file diff --git a/codev/protocols/aspir/templates/spec.md b/codev/protocols/aspir/templates/spec.md index 03c17315d..4cca2177f 100644 --- a/codev/protocols/aspir/templates/spec.md +++ b/codev/protocols/aspir/templates/spec.md @@ -152,31 +152,3 @@ Note: All consultation feedback has been incorporated directly into the relevant ## Notes [Any additional context or considerations not covered above] ---- - -## Amendments - -This section tracks all TICK amendments to this specification. TICKs are lightweight changes that refine an existing spec rather than creating a new one. - - - - \ No newline at end of file diff --git a/codev/protocols/bugfix/builder-prompt.md b/codev/protocols/bugfix/builder-prompt.md index bb950cfe4..aefa35a25 100644 --- a/codev/protocols/bugfix/builder-prompt.md +++ b/codev/protocols/bugfix/builder-prompt.md @@ -44,7 +44,7 @@ Follow the BUGFIX protocol. Read and internalize the protocol before starting an If the fix is too complex (> 300 LOC or architectural changes), notify the Architect via: ```bash -afx send architect "Issue #{{issue.number}} is more complex than expected. [Reason]. Recommend escalating to SPIR/TICK." +afx send architect "Issue #{{issue.number}} is more complex than expected. [Reason]. Recommend escalating to SPIR." ``` ## Notifications diff --git a/codev/protocols/bugfix/protocol.json b/codev/protocols/bugfix/protocol.json index cac575cbd..0abd36ee5 100644 --- a/codev/protocols/bugfix/protocol.json +++ b/codev/protocols/bugfix/protocol.json @@ -50,13 +50,11 @@ }, "build": { "command": "npm run build", - "cwd": "packages/codev", "on_fail": "retry", "max_retries": 2 }, "tests": { "command": "npm test", - "cwd": "packages/codev", "on_fail": "retry", "max_retries": 2 } diff --git a/codev/protocols/experiment/protocol.md b/codev/protocols/experiment/protocol.md index b9eabaac3..2e53487d4 100644 --- a/codev/protocols/experiment/protocol.md +++ b/codev/protocols/experiment/protocol.md @@ -10,7 +10,7 @@ Disciplined experimentation: Each experiment gets its own directory with `notes. **Use for**: Testing approaches, evaluating models, prototyping, proof-of-concept work, research spikes -**Skip for**: Production code (use SPIR), simple one-off scripts, well-understood implementations (use TICK) +**Skip for**: Production code (use SPIR), simple one-off scripts ## Structure @@ -129,11 +129,6 @@ Experiment 5 validated that [approach] achieves [results]. See: experiments/5_validation_test/notes.md ``` -### Experiment → TICK -For small, validated changes discovered during experimentation: -- Use TICK for quick implementation -- Reference experiment as justification - ## Numbering Convention Use four-digit sequential numbering (consistent with project list): diff --git a/codev/protocols/protocol-schema.json b/codev/protocols/protocol-schema.json index 375108b3d..09a994994 100644 --- a/codev/protocols/protocol-schema.json +++ b/codev/protocols/protocol-schema.json @@ -8,7 +8,7 @@ "properties": { "name": { "type": "string", - "description": "Protocol identifier (e.g., 'spir', 'tick', 'bugfix')" + "description": "Protocol identifier (e.g., 'spir', 'bugfix', 'air')" }, "version": { "type": "string", diff --git a/codev/protocols/spike/protocol.md b/codev/protocols/spike/protocol.md index 9b121d4cc..764a0bea6 100644 --- a/codev/protocols/spike/protocol.md +++ b/codev/protocols/spike/protocol.md @@ -10,7 +10,7 @@ Time-boxed technical feasibility exploration. Answer "Can we do X?" and "What wo **Use for**: Quick technical feasibility investigations, proof-of-concept explorations, "can we do X?" questions, evaluating approaches before committing to SPIR -**Skip for**: Production code (use SPIR), formal hypothesis testing (use EXPERIMENT), bug fixes (use BUGFIX), well-understood implementations (use TICK) +**Skip for**: Production code (use SPIR), formal hypothesis testing (use EXPERIMENT), bug fixes (use BUGFIX) ### Spike vs Experiment diff --git a/codev/protocols/spir/builder-prompt.md b/codev/protocols/spir/builder-prompt.md index 580e269c5..437287968 100644 --- a/codev/protocols/spir/builder-prompt.md +++ b/codev/protocols/spir/builder-prompt.md @@ -68,11 +68,34 @@ By default, the PR is opened during/after the final implement phase, with all ph The architect MAY request a PR at any point — for spec review, mid-implementation feedback, slicing a large spec into shippable PRs, etc. When the architect explicitly asks for a PR earlier (or for additional PRs), follow that direction. The prohibition is specifically on the *builder* autonomously deciding to open per-phase PRs without architect request. +### Multi-PR Mechanics (when the architect requests sequential PRs) + +Your worktree is persistent — it survives across PR merges. When the architect asks for sequential PRs (e.g., to slice a large spec into shippable pieces), use this loop: + +1. Cut a branch, open a PR, wait for merge +2. After merge: `git fetch origin && git checkout -b origin/` — where `` is the branch the architect targets PRs at (usually `main`; check the open PR's `baseRefName` if unsure) +3. Continue to the next slice, open another PR +4. Repeat + +**Important**: Do NOT run `git checkout ` — git worktrees cannot check out a branch that's checked out elsewhere. Always branch off `origin/` via fetch. + +Record PRs in status.yaml: `porch done {{project_id}} --pr --branch ` +Record merges: `porch done {{project_id}} --merged ` + +## Verify Phase + +After the final PR merges, the project enters the **verify** phase. You stay alive through verify: +1. Pull the integration branch into your worktree +2. Run `porch done {{project_id}}` to signal verification is ready +3. The architect approves `verify-approval` when satisfied + +If verification is not needed: `porch verify {{project_id}} --skip "reason"` + ## Notifications Always use `afx send architect "..."` to notify the architect at key moments: - **Gate reached**: `afx send architect "Project {{project_id}}: ready for approval"` - **PR ready**: `afx send architect "PR #N ready for review (project {{project_id}})"` -- **PR merged**: `afx send architect "Project {{project_id}} complete. PR merged. Ready for cleanup."` +- **PR merged**: `afx send architect "Project {{project_id}} PR merged. Entering verify phase."` - **Blocked**: `afx send architect "Blocked on project {{project_id}}: [reason]"` ## Handling Flaky Tests diff --git a/codev/protocols/spir/protocol.json b/codev/protocols/spir/protocol.json index 02ecb8941..0041fb5e9 100644 --- a/codev/protocols/spir/protocol.json +++ b/codev/protocols/spir/protocol.json @@ -24,7 +24,7 @@ "models": ["gemini", "codex", "claude"], "parallel": true }, - "max_iterations": 8, + "max_iterations": 3, "on_complete": { "commit": true, "push": true @@ -46,7 +46,7 @@ "models": ["gemini", "codex", "claude"], "parallel": true }, - "max_iterations": 8, + "max_iterations": 3, "on_complete": { "commit": true, "push": true @@ -73,7 +73,7 @@ "models": ["gemini", "codex", "claude"], "parallel": true }, - "max_iterations": 8, + "max_iterations": 3, "on_complete": { "commit": true, "push": true @@ -110,7 +110,7 @@ "models": ["gemini", "codex", "claude"], "parallel": true }, - "max_iterations": 8, + "max_iterations": 3, "on_complete": { "commit": true, "push": true @@ -163,7 +163,7 @@ }, "defaults": { "mode": "strict", - "max_iterations": 8, + "max_iterations": 3, "verify": { "models": ["gemini", "codex", "claude"], "parallel": true diff --git a/codev/protocols/spir/protocol.md b/codev/protocols/spir/protocol.md index f51ae04a9..5922d9883 100644 --- a/codev/protocols/spir/protocol.md +++ b/codev/protocols/spir/protocol.md @@ -1,8 +1,18 @@ # SPIR Protocol +> **SPIR** = **S**pecify → **P**lan → **I**mplement → **R**eview +> +> Each phase has one build-verify cycle with 3-way consultation. + ## Prerequisites +**Clean Worktree Before Spawning Builders**: +- All specs, plans, and local changes **MUST be committed** before `afx spawn` +- Builders work in git worktrees branched from HEAD — uncommitted files are invisible +- This includes `codev update` results, spec drafts, and plan approvals +- The `afx spawn` command enforces this (use `--force` to override) + **Required for Multi-Agent Consultation**: - The `consult` CLI must be available (installed with `npm install -g @cluesmith/codev`) - At least one consultation backend: `claude`, `gemini-cli`, or `codex` @@ -10,66 +20,6 @@ ## Protocol Configuration -### Porch Orchestration (RECOMMENDED) - -Porch is the protocol orchestrator that enforces phase transitions, manages human approval gates, and coordinates multi-agent consultations. Using porch is the recommended way to execute SPIR. - -**HOW IT WORKS:** -1. Initialize porch for your project: `porch init spir ` -2. Run the protocol loop: `porch run ` -3. Porch orchestrates Claude sessions for each phase -4. Human gates block until approved: `porch approve ` - -**STATE FILE LOCATION:** -`codev/projects/-/status.yaml` - -**PORCH COMMANDS:** -```bash -# Initialize a new SPIR project -porch init spir 0073 "user-auth" - -# Start the protocol loop (orchestrates Claude for each phase) -porch run 0073 - -# Check status -porch status 0073 - -# Approve a human gate -porch approve 0073 spec_approval - -# List pending gates across all projects -porch pending -``` - -**HUMAN GATES:** -- `spec_approval`: After specification is ready for planning -- `plan_approval`: After plan is ready for implementation - -When a gate is reached, porch notifies the architect and blocks until approved. - -**USING AF SPAWN (Architect-Builder Pattern):** -```bash -# Create worktree and start porch-driven builder (default) -afx spawn -p 0073 - -# Or for flexible, protocol-guided work (soft mode) -afx spawn --soft -p 0073 - -# Monitor builder status -afx status - -# Approve gates via dashboard or CLI -porch approve 0073 spec_approval -``` - -**MANUAL EXECUTION (without porch):** -You can still execute SPIR manually by following this document. Use signals to communicate phase completion to porch if running in orchestrated mode: -``` -SPEC_DRAFTED -PLAN_DRAFTED -PHASE_COMPLETE -``` - ### Multi-Agent Consultation (ENABLED BY DEFAULT) **DEFAULT BEHAVIOR:** @@ -101,6 +51,14 @@ The user can specify different agents by saying: "use SPIR with consultation fro ## Overview SPIR is a structured development protocol that emphasizes specification-driven development with iterative implementation and continuous review. It builds upon the DAPPER methodology with a focus on context-first development and multi-agent collaboration. +**The SPIR Model**: +- **S - Specify**: Write specification with 3-way review → Gate: `spec-approval` +- **P - Plan**: Write implementation plan with 3-way review → Gate: `plan-approval` +- **I - Implement**: Execute each plan phase with build-verify cycle (one cycle per phase) +- **R - Review**: Final review and PR preparation with 3-way review + +Each phase follows a build-verify loop: build the artifact, then verify with 3-way consultation (Gemini, Codex, Claude). + **Core Principle**: Each feature is tracked through exactly THREE documents - a specification, a plan, and a review with lessons learned - all sharing the same filename and sequential identifier. ## When to Use SPIR @@ -143,13 +101,7 @@ When filing an issue for SPIR, you can pin architectural decisions you don't wan 10. Multi-agent review of updated document 11. Final updates based on second review 12. **COMMIT**: "Final approved specification" - - Add YAML frontmatter: `approved: ` and `validated: [gemini, codex, claude]` - - Commit to the integration branch (the branch builders branch off) so builder worktrees include the artifact - - Porch will detect this metadata and skip the specify phase automatically -13. **GATE CHECK**: Before proceeding to Plan, verify all `spec_*` items complete - - If incomplete: Output "⚠️ BLOCKED" with missing items, stop - - If complete: Output "✓ Gate passed", update state, proceed -14. Iterate steps 7-12 until user approves and says to proceed to planning +13. Iterate steps 7-12 until user approves and says to proceed to planning **Important**: Keep documentation minimal - use only THREE core files with the same name: - `specs/####-descriptive-name.md` - The specification @@ -238,14 +190,7 @@ When filing an issue for SPIR, you can pin architectural decisions you don't wan 9. Multi-agent review of updated plan 10. Final updates based on second review 11. **COMMIT**: "Final approved plan" - - Add YAML frontmatter: `approved: ` and `validated: [gemini, codex, claude]` - - Commit to the integration branch (the branch builders branch off) so builder worktrees include the artifact - - Porch will detect this metadata and skip the plan phase automatically -12. **REGISTER PHASES**: For each phase defined in the plan: -13. **GATE CHECK**: Before proceeding to Implement, verify all `plan_*` items complete - - If incomplete: Output "⚠️ BLOCKED" with missing items, stop - - If complete: Output "✓ Gate passed", update state, proceed -14. Iterate steps 6-11 until agreement is reached +12. Iterate steps 6-11 until agreement is reached **Phase Design Goals**: Each phase should be: @@ -300,30 +245,21 @@ Each phase should be: **Structure**: follows the plan template provided by the plan phase **Review Required**: Yes - Technical lead approval AFTER consultations -### (IDE) - Implementation Loop - -Execute for each phase in the plan. This is a strict cycle that must be completed in order. +### I - Implement (Per Plan Phase) -**⚠️ MANDATORY**: The I-D-E cycle MUST be completed for EACH PHASE, not just at the end of all phases. Skipping D (Defend) or E (Evaluate) for any phase is a PROTOCOL VIOLATION. +Execute for each phase in the plan. Each phase follows a build-verify cycle. **CRITICAL PRECONDITION**: Before starting any phase, verify the previous phase was committed to git. No phase can begin without the prior phase's commit. -**Phase Completion Process**: -1. **Implement** - Build the code for this phase - - **GATE CHECK**: Before Defend, verify all Implement items complete -2. **Defend** - Write comprehensive tests that guard functionality - - **GATE CHECK**: Before Evaluate, verify all Defend items complete -3. **Evaluate** - Assess and discuss with user +**Build-Verify Cycle Per Phase**: +1. **Build** - Implement code and tests for this phase +2. **Verify** - 3-way consultation (Gemini, Codex, Claude) +3. **Iterate** - Address feedback until verification passes 4. **Commit** - Single atomic commit for the phase (MANDATORY before next phase) 5. **Proceed** - Move to next phase only after commit - - **GATE CHECK**: Verify all items for current phase complete - - If more phases: Proceed to next phase's Implement stage - - If last phase: Proceed to Review **Handling Failures**: -- If **Defend** phase reveals gaps → return to **Implement** to fix -- If **Evaluation** reveals unmet criteria → return to **Implement** -- If user requests changes → return to **Implement** +- If verification reveals gaps → iterate and fix - If fundamental plan flaws found → mark phase as `blocked` and revise plan **Commit Requirements**: @@ -615,19 +551,15 @@ Execute for each phase in the plan. This is a strict cycle that must be complete - Same filename as spec/plan, captures review and learnings from this feature - Methodology improvement proposals (update protocol if needed) -**FINAL GATE CHECK**: Before marking project complete, verify all `review_*` items complete -- If incomplete: Output "⚠️ BLOCKED" with missing items -- If complete: Output "✓ SPIR protocol complete for project [id]" - **Review Required**: Yes - Team retrospective recommended ## File Naming Conventions ### Specifications and Plans Format: `####-descriptive-name.md` -- Use sequential numbering (0001, 0002, etc.) +- Use sequential numbering (1, 2, etc.) - Same filename in both `specs/` and `plans/` directories -- Example: `0001-user-authentication.md` +- Example: `1-user-authentication.md` ## Status Tracking @@ -650,10 +582,10 @@ For specification/plan documents: Examples: ``` -[Spec 0001] Initial specification draft -[Spec 0001] Specification with multi-agent review -[Spec 0001] Specification with user feedback -[Spec 0001] Final approved specification +[Spec 1] Initial specification draft +[Spec 1] Specification with multi-agent review +[Spec 1] Specification with user feedback +[Spec 1] Final approved specification ``` For implementation: @@ -665,7 +597,7 @@ For implementation: Example: ``` -[Spec 0001][Phase: user-auth] feat: Add password hashing service +[Spec 1][Phase: user-auth] feat: Add password hashing service Implements bcrypt-based password hashing with configurable rounds ``` @@ -677,7 +609,7 @@ spir/####-/ Example: ``` -spir/0001-user-authentication/database-schema +spir/1-user-authentication/database-schema ``` diff --git a/codev/protocols/spir/templates/plan.md b/codev/protocols/spir/templates/plan.md index 831199848..9da106498 100644 --- a/codev/protocols/spir/templates/plan.md +++ b/codev/protocols/spir/templates/plan.md @@ -182,23 +182,3 @@ Phase 1 ──→ Phase 2 ──→ Phase 3 ## Notes [Additional context, assumptions, or considerations] ---- - -## Amendment History - -This section tracks all TICK amendments to this plan. TICKs modify both the spec and plan together as an atomic unit. - - - - \ No newline at end of file diff --git a/codev/protocols/spir/templates/spec.md b/codev/protocols/spir/templates/spec.md index 03c17315d..4cca2177f 100644 --- a/codev/protocols/spir/templates/spec.md +++ b/codev/protocols/spir/templates/spec.md @@ -152,31 +152,3 @@ Note: All consultation feedback has been incorporated directly into the relevant ## Notes [Any additional context or considerations not covered above] ---- - -## Amendments - -This section tracks all TICK amendments to this specification. TICKs are lightweight changes that refine an existing spec rather than creating a new one. - - - - \ No newline at end of file diff --git a/codev/resources/1252-behavior-baseline.md b/codev/resources/1252-behavior-baseline.md new file mode 100644 index 000000000..589b4ecf3 --- /dev/null +++ b/codev/resources/1252-behavior-baseline.md @@ -0,0 +1,166 @@ + + +# Behavioural metrics (Spec 1252, M12 / Appendix D) + +## B1 — CMAP verdict distribution + +Total verdicts: **160** across 18 projects with review history. + +| Verdict | Count | Share | +|---|---:|---:| +| REQUEST_CHANGES | 83 | 51.88% | +| APPROVE | 66 | 41.25% | +| COMMENT | 11 | 6.88% | + +**B1 REQUEST_CHANGES rate: 51.88%** — the load-bearing metric. + +## B2 — review rounds per plan phase + +mean **1.12**, median 1, max 2 (n=49 phases) + +> Advisory only. The observed range is too narrow to detect a subtle +> regression. Note also that phases advance on builder *rebuttal*, not on +> unanimous approval — 0 terminal phases in this corpus end with 3x APPROVE — +> so a "rounds to unanimity" metric would never resolve. + +## B4 — review rounds per project + +mean **3.06**, median 3 (n=18 projects). Advisory. + +## B3 — candidate scar-rule violations + +Scanned **349** files (codev/reviews + codev/state threads). +Candidate hits: **44** + +> **These are CANDIDATES, not findings.** Keyword mining cannot distinguish +> "we did this" from "never do this". Every hit requires human adjudication +> before the hard rollback trigger fires. B3 is the metric that matters most — +> it is the only one that would catch a compressed scar rule losing its force. + +| Rule | Candidate hits | +|---|---:| +| restart-tower | 21 | +| destructive-git | 9 | +| auto-approve-gate | 7 | +| kill-shellper | 6 | +| afx-from-worktree | 1 | + +### Excerpts for adjudication + +- `codev/reviews/0078-porch-e2e-testing.md:43` **[auto-approve-gate]** — 3. Auto-approve mechanism inconsistency (flag vs env var) - **NOTED**: Env var is more flexible for testing +- `codev/reviews/0078-porch-e2e-testing.md:51` **[auto-approve-gate]** — 3. Spec/plan inconsistency on auto-approve - **NOTED**: Env var approach is valid +- `codev/reviews/0078-porch-e2e-testing.md:56` **[auto-approve-gate]** — Chose `PORCH_AUTO_APPROVE` environment variable over `--auto-approve-gates` flag because: +- `codev/reviews/0078-porch-e2e-testing.md:79` **[auto-approve-gate]** — | Full Lifecycle | ✅ | End-to-end with auto-approve | +- `codev/reviews/0116-shellper-resource-leakage.md:25` **[kill-shellper]** — - E2E test: Tower periodic cleanup timer removes stale sockets during runtime (2s interval, externally killed shellper) +- `codev/reviews/0118-shellper-multi-client.md:41` **[kill-shellper]** — - **Removed stale tests from session-manager.test.ts**: Two tests were removed (`repeated calls are idempotent` and `kills child process when readShellperInfo fails`). These were fragile timing-based +- `codev/reviews/1096-vscode-codev-openissuebynumber.md:53` **[restart-tower]** — - Manual verification: the human approved the running worktree at the `dev-approval` gate. Note the GitHub browser-open path requires the running Tower to serve the updated `issue-view` script (rebuil +- `codev/reviews/1096-vscode-codev-openissuebynumber.md:99` **[restart-tower]** — from the worktree root, `pnpm build && pnpm -w run local-install` (rebuilds + restarts Tower with the new `issue-view` script), then reload the VSCode window. +- `codev/reviews/1118-consolidate-state-db-tables-in.md:144` **[afx-from-worktree]** — Nesting is an unsupported anti-pattern (`afx spawn` from inside a worktree), so this is a +- `codev/reviews/1140-afx-workspace-recover-respawne.md:52` **[kill-shellper]** — 2. Kill that builder's shellper (or reboot), run `afx workspace recover --apply` from main's terminal +- `codev/reviews/1145-codev-adopt-launchinstance-mai.md:73` **[restart-tower]** — - **Run locally**: `pnpm build && pnpm -w run local-install` (restarts Tower) +- `codev/reviews/1149-tower-recovery-claude-architec.md:59` **[restart-tower]** — - **Run**: `pnpm build && pnpm -w run local-install` (restarts Tower with this branch) +- `codev/reviews/1149-tower-recovery-claude-architec.md:63` **[kill-shellper]** — 3. Kill the architect's claude process (not the shellper) +- `codev/reviews/1149-tower-recovery-claude-architec.md:66` **[restart-tower]** — 6. Escape hatch: repeat the corruption, restart Tower with `CODEV_SKIP_RESUME=1`, verify the architect starts fresh with no crash cycle at all +- `codev/reviews/438-aspir-protocol.md:31` **[auto-approve-gate]** — - **Phase 3 documentation wording**: The plan suggested specific bullet points for the CLAUDE.md/AGENTS.md ASPIR section. During implementation, the root CLAUDE.md was written first with slightly diff +- `codev/reviews/438-aspir-protocol.md:127` **[auto-approve-gate]** — - **Concern**: CLAUDE.md says gates are "auto-approved" which contradicts protocol.json (gates are absent) +- `codev/reviews/841-feat-vscode-architects-tree-ad.md:63` **[destructive-git]** — None skipped. Note (not flaky — pre-existing and unrelated): `packages/vscode/src/__tests__/terminal-adapter.test.ts` and `reconnect-link-provider.test.ts` fail to *collect* under plain vitest (dynami +- `codev/reviews/883-vscode-builder-cleanup-no-long.md:107` **[restart-tower]** — - **Build + install**: `pnpm build && pnpm -w run local-install` (restarts Tower, picks up the patched extension) +- `codev/reviews/916-vscode-sidebar-data-builders-b.md:116` **[restart-tower]** — - Simulate a transient blip (stop/restart Tower, or toggle network so the SSE drops then reconnects): +- `codev/reviews/920-vscode-editor-tab-webview-for-.md:83` **[restart-tower]** — 1. `cd .builders/pir-920 && pnpm -w run local-install` — rebuilds + restarts Tower so it serves `/api/issue-search` and resolves the `issue-search` concept. +- `codev/reviews/930-vscode-mark-recently-created-b.md:67` **[destructive-git]** — behavior (confirmed via `git stash` that the `status.ts`/`workspace.ts`/ +- `codev/reviews/961-core-extract-transport-agnosti.md:73` **[restart-tower]** — - Restart Tower; confirm a fresh connect resets the counter on both surfaces. +- `codev/reviews/982-vscode-tower-no-active-termina.md:59` **[kill-shellper]** — - Persistent: kill a builder's shellper so its session can't reconnect → after the retries, the actionable toast appears; **Retry** re-attempts, **Recover Builders** opens a terminal running `afx work +- `codev/reviews/983-vscode-tower-detect-installed-.md:7` **[restart-tower]** — The v3.1.7 #791 preflight verifies the *installed* `codev` CLI version but is blind to whether the *running* Tower process is executing that same code — after an `npm install -g` upgrade without a Tow +- `codev/reviews/983-vscode-tower-detect-installed-.md:59` **[restart-tower]** — - **Codex REQUEST_CHANGES (3-way consult, addressed)** — Codex (HIGH confidence) flagged that the `too-old` (404) path had the same futile-remedy bug we'd already fixed for the ext-version comparison: +- `codev/reviews/983-vscode-tower-detect-installed-.md:63` **[restart-tower]** — - **Self-invoked restart safety** — the in-extension `Restart Tower` action is only safe because #991 (already on main) scoped `afx tower stop` to the listening Tower process; before it, the unfiltere +- `codev/reviews/983-vscode-tower-detect-installed-.md:73` **[restart-tower]** — - Stale: bump the globally-installed `@cluesmith/codev/package.json` version (simulates an upgrade) without restarting Tower, reload the Ext Dev Host → divergence toast with `Restart Tower`; clicking +- `codev/reviews/983-vscode-tower-detect-installed-.md:75` **[restart-tower]** — - Unreachable: stop Tower → only the existing "not connected" path, no new toast +- `codev/reviews/991-terminal-stale-tab-on-a-pre-re.md:67` **[restart-tower]** — - **Deploy ordering:** the host-kill fix lives in `afx tower stop`; the id-preservation lives in the Tower server. Both ship via `local-install`, which restarts Tower using the freshly-installed `afx` +- `codev/reviews/997-tower-reconcile-terminal-sessi.md:58` **[restart-tower]** — - **Run / verify** (this repo restarts Tower to pick up the change): +- `codev/reviews/997-tower-reconcile-terminal-sessi.md:59` **[restart-tower]** — 1. `pnpm -w run local-install` with a builder/architect terminal active (rebuilds + restarts Tower) +- `codev/state/air-1238_thread.md:67` **[restart-tower]** — Did NOT restart Tower to observe the startup sweep live — that would kill every running +- `codev/state/aspir-1210_thread.md:86` **[auto-approve-gate]** — - **STOPPED at pr gate — awaiting human approval.** Not auto-approving (human-only gate). Architect +- `codev/state/bugfix-1122_thread.md:21` **[destructive-git]** — While proving the regression test fails without the fix, I ran a `git stash push` +- `codev/state/bugfix-1122_thread.md:23` **[destructive-git]** — then a paired `git stash pop` — which popped the PRE-EXISTING `stash@{0}: On main: +- `codev/state/bugfix-1122_thread.md:35` **[destructive-git]** — - `git reset --hard HEAD` reverted the 12 tracked modifications and removed the 32 +- `codev/state/bugfix-1122_thread.md:39` **[destructive-git]** — - Verified `git stash list` still shows `stash@{0}` intact afterward. +- `codev/state/bugfix-1224_thread.md:104` **[kill-shellper]** — - SessionManager maxRestarts give-up: logStderrTail (capture child reason) + kill shellper group +- `codev/state/bugfix-838_thread.md:36` **[destructive-git]** — Both fail against pre-fix package.json (verified via `git stash`); both +- `codev/state/pir-1118_thread.md:158` **[destructive-git]** — (a git stash pop unstaged it); committed the deletion (14604d24). CI now GREEN. +- `codev/state/pir-933_thread.md:11` **[destructive-git]** — - `git reset --hard fcea5028` (pristine porch-init commit) → porch back in the +- `codev/state/pir-983_thread.md:34` **[restart-tower]** — - **vscode**: `decideTowerStatus` + `towerDivergenceMessage` (pure, in preflight-core, 8 new unit tests); `probeTowerVersion` fired on each `connected` transition in `extension.ts`; dedicated `showTow +- `codev/state/pir-983_thread.md:48` **[restart-tower]** — **Rebased on main (picked up PIR #991 / PR #999).** Clean rebase, no conflicts. Re-verified every line ref in the plan — all still accurate (RouteContext tower-routes.ts:131, routeCtx tower-server.ts: +- `codev/state/pir-991_thread.md:66` **[restart-tower]** — Pending reviewer re-test with a freshly-built vsix (reload window once, then restart Tower only): expect Codev output `Tower reconnected — re-syncing N terminal(s)` → `recoverSuccessor(...): poll 1/5 + +## B5 — consult cost/duration + +**Not captured here.** `consult stats` is a rolling 30-day machine-local DB, +so it is not reproducible from a commit. Capture it separately as advisory +context; it is excluded from T14 determinism and drives no rollback trigger. + +## Sample provenance + +- 0092-terminal-file-links +- 0120-codex-sdk-integration +- 0124-test-suite-consolidation +- 403-af-send-typing-awareness +- 456-dashboard-statistics-tab-in-ri +- 462-add-spike-protocol-for-technic +- 467-add-open-files-shells-section- +- 468-af-rename-command-to-rename-cu +- 589-support-non-github-repositorie +- 723-improve-arch-md-lessons-learne +- 746-spir-architect-s-baked-archite +- 755-multi-architect-support-per-ar +- 761-surface-multiple-architects-in +- 778-gemini-cli-antigravity-cli-jun +- 786-multi-architect-feature-is-und +- 823-multi-architect-coordination-b +- 927-needs-attention-surface-prs-vi +- 987-engineering-wisdom-is-write-on + + +--- + +## B5 — consult cost/duration: forward snapshot (advisory) + +**Captured 2026-07-27 via `consult stats --json` (rolling 30-day machine-local +DB). NON-DETERMINISTIC: this section is appended at capture time, not +regenerated by the measurement script, and it is excluded from T14's +determinism assertion and from every rollback trigger.** It exists as advisory +context for interpreting a B1 movement in the verify phase — e.g. a REQUEST_CHANGES +rise accompanied by a duration collapse suggests degraded review quality rather +than degraded prompts. + +| Model | Calls (30d) | Avg duration | Total cost | Success rate | +|---|---:|---:|---:|---:| +| gemini | 1272 | 23.3s | n/a (no cost data) | 89.8% | +| claude | 971 | 107.6s | $900.06 | 91.9% | +| codex | 875 | 77.5s | $577.95 | 87.9% | +| **total** | **3349** | — | **$1478.01** (1278 of 3349 with cost data) | 90.6% | + +Verify-phase comparison: capture the same snapshot at verify time and compare +per-model averages directionally only. No threshold keys off this table. diff --git a/codev/resources/1252-shadow-tree-audit.md b/codev/resources/1252-shadow-tree-audit.md new file mode 100644 index 000000000..65a98f2b8 --- /dev/null +++ b/codev/resources/1252-shadow-tree-audit.md @@ -0,0 +1,282 @@ +# Spec 1252 — Shadow-tree local-unique content audit (M11) + +> **HARVEST NOTE (2026-07-28).** This audit was produced on branch +> `builder/spir-1252` (PR #1278, closed unmerged — see the 1252 review). The +> enforcement machinery it references (T11/shadow-drift gate, deletion, +> equivalence tests) lives on that branch and did NOT merge; what merged is +> the 17-file content reconciliation this audit justified, plus the knowledge +> artifacts. The shadow tree still exists on main by deliberate decision — +> the successor project re-decides its fate after shrinking the surface. + + + + +## Count correction vs the spec + +The spec said **76** shadow copies. The mechanical enumeration finds **77**: +the spec's count missed `consult-types/integration-review.md` (the drift +audit's `FRAMEWORK_DRIFT_DIRS` covers `protocols`, `roles`, AND +`consult-types`). 73 protocols-tree files (63 .md + 10 .json), 3 roles, 1 +consult-types. Drifted: 17, matching the spec. + +## Classification method (D1/D2) + +- **rot** — local lags the skeleton (missing content, stale references, + superseded structure). Evidence recorded per row. → TS1: reconcile to + skeleton (Phase 3), delete with the tree (Phase 4). +- **local-unique** — content present locally, absent from the skeleton, that + plausibly encodes codev-specific functionality. → **escalated to the + architect; nothing overwritten or deleted until ruled** (blocking for that + file only). +- Ambiguity defaults to local-unique/escalate per D2. + +Every hunk of each drifted file was classified, not just the file as a whole +(a local-unique paragraph inside an otherwise-rotted file is exactly the loss +M11 exists to prevent). Key evidence used: + +- **TICK is retired**: no `tick/` protocol dir exists in either tree; porch + source references it only in one stale test; the porch skill text naming + TICK is stale. All TICK references in local copies are therefore rot. (But + note: the skeleton's own porch phase prompts still emit TICK-amendment + language — upstream inconsistency, flagged for Phase 8's governance sweep.) +- **Obsolete gate IDs**: local `spir/protocol.md`'s retained porch section uses + `spec_approval` (underscore) — porch's actual gate is `spec-approval`. The + "local additions" predate the skeleton's restructure; they are what the + skeleton deleted, not new content. +- **architect.md**: the skeleton *generalized* the labels section to + `/` placeholders for adopters; the codev-specific `area/` recipes the + local copy hardcodes are already owned by CLAUDE.md (single-owner rule), so + taking the skeleton loses nothing. + +## Audit table (77 rows) + +| # | File | Divergence | Classification | Terminal state | Ruling / note | +|---|---|---|---|---|---| +| 1 | `consult-types/integration-review.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 2 | `protocols/air/builder-prompt.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 3 | `protocols/air/consult-types/impl-review.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 4 | `protocols/air/consult-types/pr-review.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 5 | `protocols/air/prompts/implement.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 6 | `protocols/air/prompts/pr.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 7 | `protocols/air/protocol.json` | identical | local-unique | TS1 | RULED TS1 (architect, 2026-07-28): cwd migrated to .codev/config.json porch.checks (all four protocols); skeleton json taken | +| 8 | `protocols/air/protocol.md` | identical | rot | TS1 | RECONCILED Phase 3 (was: 1-line delta; skeleton wording current) | +| 9 | `protocols/aspir/builder-prompt.md` | identical | rot | TS1 | RECONCILED Phase 3 (was: missing Multi-PR Mechanics + Verify Phase sections (skeleton has them)) | +| 10 | `protocols/aspir/consult-types/impl-review.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 11 | `protocols/aspir/consult-types/phase-review.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 12 | `protocols/aspir/consult-types/plan-review.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 13 | `protocols/aspir/consult-types/pr-review.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 14 | `protocols/aspir/consult-types/spec-review.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 15 | `protocols/aspir/prompts/implement.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 16 | `protocols/aspir/prompts/plan.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 17 | `protocols/aspir/prompts/review.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 18 | `protocols/aspir/prompts/specify.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 19 | `protocols/aspir/protocol.json` | identical | local-unique | TS1 | RULED TS1 (architect, 2026-07-28): same ruling as spir | +| 20 | `protocols/aspir/protocol.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 21 | `protocols/aspir/templates/plan.md` | identical | rot | TS1 | RECONCILED Phase 3 (was: local-only TICK Amendments section; TICK retired (no protocol dir in either tree; porch refs only in one stale test)) | +| 22 | `protocols/aspir/templates/review.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 23 | `protocols/aspir/templates/spec.md` | identical | rot | TS1 | RECONCILED Phase 3 (was: same TICK Amendments section) | +| 24 | `protocols/bugfix/builder-prompt.md` | identical | rot | TS1 | RECONCILED Phase 3 (was: local says SPIR/TICK; TICK retired) | +| 25 | `protocols/bugfix/consult-types/impl-review.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 26 | `protocols/bugfix/consult-types/pr-review.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 27 | `protocols/bugfix/prompts/fix.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 28 | `protocols/bugfix/prompts/investigate.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 29 | `protocols/bugfix/prompts/pr.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 30 | `protocols/bugfix/protocol.json` | identical | local-unique | TS1 | RULED TS1 (architect, 2026-07-28): same ruling as air | +| 31 | `protocols/bugfix/protocol.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 32 | `protocols/experiment/builder-prompt.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 33 | `protocols/experiment/protocol.json` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 34 | `protocols/experiment/protocol.md` | identical | rot | TS1 | RECONCILED Phase 3 (was: local TICK handoff section; TICK retired) | +| 35 | `protocols/experiment/templates/notes.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 36 | `protocols/maintain/builder-prompt.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 37 | `protocols/maintain/consult-types/impl-review.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 38 | `protocols/maintain/consult-types/pr-review.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 39 | `protocols/maintain/prompts/maintain.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 40 | `protocols/maintain/prompts/review.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 41 | `protocols/maintain/protocol.json` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 42 | `protocols/maintain/protocol.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 43 | `protocols/maintain/templates/maintenance-run.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 44 | `protocols/pir/builder-prompt.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 45 | `protocols/pir/consult-types/impl-review.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 46 | `protocols/pir/consult-types/pr-review.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 47 | `protocols/pir/prompts/implement.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 48 | `protocols/pir/prompts/plan.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 49 | `protocols/pir/prompts/review.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 50 | `protocols/pir/protocol.json` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 51 | `protocols/pir/protocol.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 52 | `protocols/protocol-schema.json` | identical | rot | TS1 | RECONCILED Phase 3 (was: local example lists tick; TICK retired) | +| 53 | `protocols/research/builder-prompt.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 54 | `protocols/research/protocol.json` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 55 | `protocols/research/protocol.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 56 | `protocols/spike/builder-prompt.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 57 | `protocols/spike/protocol.json` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 58 | `protocols/spike/protocol.md` | identical | rot | TS1 | RECONCILED Phase 3 (was: local mentions TICK; TICK retired) | +| 59 | `protocols/spike/templates/findings.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 60 | `protocols/spir/builder-prompt.md` | identical | rot | TS1 | RECONCILED Phase 3 (was: THE HEADLINE DRIFT: local missing Multi-PR Mechanics + Verify Phase + "Entering verify phase." string) | +| 61 | `protocols/spir/consult-types/impl-review.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 62 | `protocols/spir/consult-types/phase-review.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 63 | `protocols/spir/consult-types/plan-review.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 64 | `protocols/spir/consult-types/pr-review.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 65 | `protocols/spir/consult-types/spec-review.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 66 | `protocols/spir/prompts/implement.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 67 | `protocols/spir/prompts/plan.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 68 | `protocols/spir/prompts/review.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 69 | `protocols/spir/prompts/specify.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 70 | `protocols/spir/protocol.json` | identical | local-unique | TS1 | RULED TS1 (architect, 2026-07-28): accept skeleton max_iterations 3 — B2 evidence (rounds 1-2, mean 1.12) persuasive; >3 rounds is a human-attention moment | +| 71 | `protocols/spir/protocol.md` | identical | rot | TS1 | RECONCILED Phase 3 (was: local keeps pre-restructure porch section with obsolete underscore gate IDs (spec_approval); skeleton moved to build-verify model) | +| 72 | `protocols/spir/templates/plan.md` | identical | rot | TS1 | RECONCILED Phase 3 (was: local-only TICK Amendments section; TICK retired) | +| 73 | `protocols/spir/templates/review.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 74 | `protocols/spir/templates/spec.md` | identical | rot | TS1 | RECONCILED Phase 3 (was: same TICK Amendments section) | +| 75 | `roles/architect.md` | identical | rot | TS1 | RECONCILED Phase 3 (was: skeleton generalized labels section to / placeholders; codev-specific area/ recipes already owned by CLAUDE.md — no content lost) | +| 76 | `roles/builder.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +| 77 | `roles/consultant.md` | identical | rot | TS1 | byte-identical redundant copy; nothing to lose | +## Local-only entries (NOT shadow copies — preserved, T8) + +| File | Why it survives | +|---|---| +| `codev/protocols/release/` (whole dir) | Genuine local-only protocol; no skeleton counterpart | +| `codev/protocols/maintain/templates/audit-report.md` | Local-only template; no skeleton counterpart | +| `codev/protocols/maintain/templates/lessons-learned.md` | Local-only template; no skeleton counterpart | + +## Open escalations (4 files, 2 questions) + +Sent to the architect via afx send (Phase 2). Blocking those 4 files only; +the 13 rot files proceed to Phase 3 reconciliation. + +1. **max_iterations 8 (local) vs 3 (skeleton)** — `spir/protocol.json`, + `aspir/protocol.json`. Behavioral: bounds the CMAP iterate-until-approve + loop. No config override exists for this field. Options: accept skeleton's 3 + (TS1); retain forked json codev-only (TS3, allowlisted); promote 8 into the + skeleton (TS2). +2. **`"cwd": "packages/codev"` in check commands** — `air/protocol.json`, + `bugfix/protocol.json`. Functional: monorepo check cwd. The sanctioned + mechanism is `.codev/config.json` → `porch.checks` cwd overrides (Spec + #550), which this repo does not currently use. Recommended: migrate cwd to + config (making spir/aspir/air/bugfix consistent), then take skeleton json — + TS1-after-migration. Note spir/aspir currently pass only because the repo + root's package.json happens to have build/test scripts. + +## M7 compatibility audit — COMPLETE (Phase 4, step 4a) + +Repo-wide classification of every literal `codev/protocols/` / `codev/roles/` +reference. Production consumers were audited in spec Appendix B (all +resolver-routed or inert). This completes the test tier and non-TypeScript +consumers, as M7 requires before any deletion. + +### Test tier — 4 real repo-readers (MUST repoint at `codev-skeleton/` in the deletion commit) + +| Test | What it reads | Fix | +|---|---|---| +| `baked-decisions.test.ts` | `codev/protocols/{spir,aspir,air}/builder-prompt.md` + prompt files (Spec 746 PHASE lists) | repoint `relPath` entries to `codev-skeleton/...`; skeleton mirror entries collapse to the same path | +| `bugfix-742-consult-templates.test.ts` | 4 consult-type files under `codev/protocols/{bugfix,spir}` | repoint `repoRoot` joins to `codev-skeleton/` | +| `protocol-prompt-audit.test.ts` | walks `codev/protocols/` wholesale | walk `codev-skeleton/protocols/` | +| `bugfix-685-close-keyword.test.ts` | 6 prompt files across protocols | repoint table to `codev-skeleton/` | + +After deletion, `codev-skeleton/` is the single in-repo source (the installed +skeleton is its build-time copy), so repointing keeps these tests validating +the same content agents are served — no assertion weakens. + +### Test tier — safe (temp fixtures or string literals only) + +`consult.test.ts`, `pr-gate-audit.test.ts` (both build `codev/` trees inside +temp roots), `porch/protocol.test.ts`, `protocol-overrides.test.ts`, +`roles.test.ts`, `spawn-roles.test.ts`, `af-architect.test.ts`, +`tower-utils.test.ts`, `bugfix-527`, `bugfix-744`, `bugfix-619`, +`governance-sweep.test.ts` (strings/paths in messages only), +`framework-ref-audit.test.ts` (scans the skeleton by design). + +### Non-TypeScript consumers + +| File | Kind | Deletion-safe? | +|---|---|---| +| `scripts/measure-prompt-surface.sh` | two-tier resolve (`codev/` then `codev-skeleton/`) — this project's own script, built deletion-aware | **Yes** — falls through to skeleton | +| `apps/vscode/scripts/publish.sh:15` | comment referencing `codev/protocols/release/` | **Yes** — and `release/` is preserved anyway | + +No other shell/yaml/json consumer found outside `node_modules`/`dist`/worktrees. + +### Verdict + +Deletion is safe once (a) the four repo-reading tests are repointed in the same +commit, and (b) the four ESCALATED json rulings resolve. No production code +path reads the shadow tree by literal path. + +## Escalation rulings (architect, 2026-07-28) + +- **E1 → TS1**: skeleton's `max_iterations: 3` accepted for spir + aspir. The + B2 baseline evidence (rounds run 1–2, mean 1.12) was the deciding factor — + ">3 rounds is a human-attention moment, not an automation moment." +- **E2 → migrate-then-TS1**: `cwd` moved to `.codev/config.json` → + `porch.checks` for **all four** protocols; skeleton jsons taken. + +## E2 config-reach verification (required by the ruling) + +`.codev/config.json` is untracked (`.gitignore:11`), so the ruling required +verifying the config reaches every context that runs porch checks before +classifying E2 resolved. Verified against code, not assumption: + +| Context | Mechanism | Verified | +|---|---|---| +| Builder worktrees (future spawns) | `spawn-worktree.ts:100–113` symlinks main's `.codev/config.json` into every new worktree when it exists | code read | +| Builder worktrees (existing) | `afx setup ` re-applies symlinks; independently covered by the fallback below | docs + fallback | +| Main checkout | file must be created there once (snippet sent to the architect — a builder cannot write outside its worktree) | fallback covers the gap meanwhile | +| Fresh clone (no config anywhere) | porch runs checks at the workspace root; **root `package.json` delegates** `build`/`test` to `packages/codev` via pnpm filters | **empirically proven** — spir/aspir checks ran cwd-less at this worktree's root, green, all session | +| CI | `.github/workflows/test.yml` never invokes porch checks (it only excludes porch e2e dirs from a vitest run) | grep of workflows | + +**Key finding: the migration is defense-in-depth, not load-bearing.** Every +context has a working fallback (root delegation) even with no config file +anywhere, so no context can break. No escalate-back needed. This worktree's +`.codev/config.json` now carries the overrides +(`build/test/e2e_tests/build_succeeds/tests_pass → cwd: packages/codev`). + +## M7 correction (Phase 4b): 8 repo-reading tests, not 4 + +The step-4a classification used a path-construction regex and had **four false +negatives** — tests that read `codev/protocols/*` via `path.join` segments, +template literals, or `__dirname` traversal that the regex missed: +`bugfix-744-spir-pr-strategy`, `governance-sweep`, `review-prompt-routing` +(iterated both trees), and `bugfix-619-aspir-prompt`. The deletion itself +caught them (ENOENT), which is the correct failure mode — loud, at the point +of change. All eight are now repointed at `codev-skeleton/`; mirror-parity +constructs collapsed to skeleton-only with explanatory comments. Lesson for +the review: a grep classification of consumers is a hypothesis, not a proof — +the deletion is the test. + +## Phase 4 result + +- 77 shadow copies deleted; `codev/roles/` and `codev/consult-types/` removed + entirely; `codev/protocols/` retains only the three preserved local-only + files (`release/protocol.md`, `maintain/templates/{audit-report,lessons-learned}.md`). +- `copyRoles`/`copyProtocols` (124 lines) removed from `scaffold.ts` with a + tombstone comment; `copyRoles` test block removed. +- M10 equivalence: `shadow-removal-manifest.json` (77 pre-deletion sha256s) + + `shadow-removal-equivalence.test.ts` prove the resolver serves byte-identical + content post-deletion FROM THE SKELETON TIER (path asserted, not just + content), and assembled-prompt equivalence is exercised directly: the real + `buildPromptFromTemplate` ran against the pre-deletion tree (git archive) for + all nine protocols under a fixed context, snapshots committed, and the live + assembly must reproduce them byte-for-byte. The FULL served prompt (preamble + + template) is additionally guarded: the spawn preamble had instructed + "Read codev/roles/builder.md" — a literal-path fetch of a deleted file + (caught by Codex, Phase 4 iter-4) — now replaced by a single-owner + `builderPreamble()` that points at the harness-injected role + (`.builder-role.md`), with tests asserting no framework path is fetched. +- Allowlist: **empty**. Drift gate: zero findings. All T11 guards green with + zero pending rows. + +## Phase 4 iter-2 additions + +- `codev/protocols/release/protocol.md:43` repointed (`cat codev-skeleton/...`) + — the M7 sweep had not audited codev/'s own preserved content. Verified no + other fetch of deleted paths exists in preserved files. +- **Deferred to Phase 8 (recorded so it is not lost)**: `CLAUDE.md`/`AGENTS.md` + contain ~10 prose references to deleted `codev/protocols/...` paths + (available-protocol listing, RELEASE pointer, SPIR instruction). References, + not fetches; both files receive Phase 5/7/8 edits where these repoint to + `codev-skeleton/` (or resolver-neutral wording) as part of governance sync. diff --git a/codev/resources/1252-word-after-phase7.md b/codev/resources/1252-word-after-phase7.md new file mode 100644 index 000000000..f36be3fe7 --- /dev/null +++ b/codev/resources/1252-word-after-phase7.md @@ -0,0 +1,58 @@ + +# Prompt-surface measurement + +Commit: `883b3a8a` + +## Per-surface word counts + +| Surface | Words | Files | Load | +|---|---:|---:|---| +| CLAUDE.md | 4330 | 1 | always-on (hot tier inlined) | +| AGENTS.md | 4330 | 1 | always-on twin (other tools) | +| arch-critical.md (HOT) | 422 | 1 | inlined in CLAUDE.md + every phase prompt | +| lessons-critical.md (HOT) | 320 | 1 | same | +| arch.md (COLD) | 20240 | 1 | on demand | +| lessons-learned.md (COLD) | 21270 | 1 | on demand | +| codev-skeleton/** | 78804 | 123 | resolver tier 4 | +| — skeleton protocols/ | 43812 | — | | +| — skeleton roles/ | 3836 | — | | +| codev/protocols/ (shadow) | 2347 | 3 | wins over skeleton | +| codev/roles/ (shadow) | 0 | 0 | wins over skeleton | +| .claude/skills/** | 16769 | 14 | on demand | + +## Always-on load per builder (derived from resolved artifacts) + + CLAUDE.md 4330 + + spawn prompt (proxy) 4564 (builder-prompt.md 861 + protocol.md 3703) + + phase task (proxy) x 10 11430 (hot tier 742 + porch prompt mean 401, per iteration) + ------------------------------ + = ALWAYS_ON 20324 + +Proxies deliberately exclude per-project variable content (issue body, +task-JSON boilerplate) — those are not trimmable prompt surface, and including +them would let a rerun's delta be polluted by whichever issue happened to spawn +the measuring builder. AGENTS.md and the hot-tier files are likewise excluded +from the top-level sum: AGENTS.md is the byte-identical twin (one loads per +session, not both), and the hot-tier files appear inside CLAUDE.md's count +(they ARE counted once per phase task, where porch injects them separately). + +ALWAYS_ON_WORDS=20324 + +## N1 delta vs baseline (required record) + +| | Baseline (Phase 1, `1252-word-baseline.md`) | After Phase 7 | Delta | +|---|---:|---:|---:| +| CLAUDE.md | 5,773 | 4,330 | −1,443 | +| Spawn proxy (served) | 4,723 | 4,564 | −159 | +| Phase task ×10 | 11,360 | 11,430 | +70 | +| **ALWAYS_ON** | **21,856** | **20,324** | **−1,532** | + +**Achieved: −7.0% against the ≥20% target. N1 is a target, not a gate; this is +the honest figure.** Decomposition: CLAUDE.md relocations/trims delivered the +bulk; drift repair *added* served words the prompts were supposed to have +(Multi-PR + Verify, ~207); D3's promotion of three user-global scar rules +added ~120; the remaining always-on surface is single-owned prose whose +largest block (protocol.md, 3,703 words) is protocol semantics — excluded by +the spec's Non-goals. The 20% arithmetic assumed more of the surface was +duplicated rules than measurement bore out. Whether even −7% moved behaviour +is the verify phase's question (M12b). diff --git a/codev/resources/1252-word-baseline.md b/codev/resources/1252-word-baseline.md new file mode 100644 index 000000000..99aa8abe5 --- /dev/null +++ b/codev/resources/1252-word-baseline.md @@ -0,0 +1,39 @@ + +# Prompt-surface measurement + +Commit: `b37f7db1` + +## Per-surface word counts + +| Surface | Words | Files | Load | +|---|---:|---:|---| +| CLAUDE.md | 5773 | 1 | always-on (hot tier inlined) | +| AGENTS.md | 5773 | 1 | always-on twin (other tools) | +| arch-critical.md (HOT) | 416 | 1 | inlined in CLAUDE.md + every phase prompt | +| lessons-critical.md (HOT) | 320 | 1 | same | +| arch.md (COLD) | 20240 | 1 | on demand | +| lessons-learned.md (COLD) | 21270 | 1 | on demand | +| codev-skeleton/** | 77955 | 113 | resolver tier 4 | +| — skeleton protocols/ | 44784 | — | | +| — skeleton roles/ | 3813 | — | | +| codev/protocols/ (shadow) | 47511 | 66 | wins over skeleton | +| codev/roles/ (shadow) | 3687 | 3 | wins over skeleton | +| .claude/skills/** | 16743 | 14 | on demand | + +## Always-on load per builder (derived from resolved artifacts) + + CLAUDE.md 5773 + + spawn prompt (proxy) 4723 (builder-prompt.md 636 + protocol.md 4087) + + phase task (proxy) x 10 11360 (hot tier 736 + porch prompt mean 400, per iteration) + ------------------------------ + = ALWAYS_ON 21856 + +Proxies deliberately exclude per-project variable content (issue body, +task-JSON boilerplate) — those are not trimmable prompt surface, and including +them would let a rerun's delta be polluted by whichever issue happened to spawn +the measuring builder. AGENTS.md and the hot-tier files are likewise excluded +from the top-level sum: AGENTS.md is the byte-identical twin (one loads per +session, not both), and the hot-tier files appear inside CLAUDE.md's count +(they ARE counted once per phase task, where porch injects them separately). + +ALWAYS_ON_WORDS=21856 diff --git a/codev/reviews/1252-prompt-architecture-single-own.md b/codev/reviews/1252-prompt-architecture-single-own.md new file mode 100644 index 000000000..835a3ef78 --- /dev/null +++ b/codev/reviews/1252-prompt-architecture-single-own.md @@ -0,0 +1,152 @@ +# Review 1252: Prompt Architecture — Single-Owner Rule for Instruction Content + +## Outcome: harvested, not merged + +The full implementation (8 phases, unanimous CMAP approval at each phase's +final iteration, PR #1278) reached the pr gate and was **deliberately not +merged**. Waleed's charter-level call at the gate: the structural machinery +(shadow-tree deletion, scar registry + CI enforcement, ownership map, +dedup partials) is not worth carrying for *this* surface, because a successor +project will first shrink the surface radically (blog-principles judgment +rewrite; >50% reduction target; frontier-models-only assumption) and +enforcement should be rebuilt fit-for-purpose around the post-shrink surface. +This was a risk/reward re-scope, not a quality verdict — the audit and +measurement work from this project directly shaped the successor's charter. + +**What landed on main (this harvest PR):** + +- **The 17-file drift reconciliation** — the one live bug. `codev/`'s shadow + copies synced to the skeleton content, restoring the served SPIR/ASPIR + builder prompts' `Multi-PR Mechanics` and `Verify Phase` sections and the + `"Entering verify phase."` notification; architect rulings E1 + (`max_iterations: 3`, decided on the B2 evidence that review rounds run + 1–2) and E2 (check `cwd`s moved to main's `.codev/config.json` + `porch.checks`; the forked protocol.json files synced) applied. Spec-746 + pure-addition baselines re-derived for the synced files (a frozen-file + baseline breaks on any legitimate edit — see Lessons). +- **The knowledge artifacts**: `1252-shadow-tree-audit.md` (the 77-row + classification with rulings), `1252-word-baseline.md` / + `1252-word-after-phase7.md` (M6/N1 record), `1252-behavior-baseline.md` + (M12a — B1 51.88% REQUEST_CHANGES over 160 verdicts, with the B5 snapshot), + the measurement tooling (`scripts/measure-prompt-surface.sh`, + `packages/codev/src/lib/prompt-behavior-metrics.ts` + CLI + its + determinism test), this review, and the builder thread + (`codev/state/spir-1252_thread.md`). + +**What stayed on the branch** (`builder/spir-1252`, PR #1278 closed unmerged, +preserved for reference — do not delete): shadow-tree deletion + equivalence +proofs, scar registry + line-exact T6 enforcement, ownership map + T12 +completeness machinery, dedup partials + served-surface guard, CLAUDE.md +restructure, all skeleton edits, and their ~15 test suites. **The scar-registry +concept and the eight ratified rule wordings move into the successor charter** +— they are deferred, not discarded. + +## Summary of the full implementation (for the record) + +Issue #1252 asked for a single-owner rule with an inventory before +restructuring. Measurement reshaped the problem: the largest duplication axis +was a 77-file shadow tree under `codev/protocols|roles|consult-types` that +outranked the installed skeleton in the resolver and had drifted in 17 files — +including the served SPIR builder prompt, missing its entire `Verify Phase` +section. The project filed to fix prompt drift was itself spawned with a +drifted prompt, and the existing detector (#1210) had been reporting the +problem to `codev doctor`, unread, the whole time. + +The eight phases (all on the reference branch): drift CI gate + baselines; +77-row local-unique audit with TS1–TS4 terminal states and 4 architect-ruled +escalations; the 17-file reconciliation; shadow-tree deletion with +three-layer equivalence proof (resolver tier, byte-identical assembly across +nine protocols, spawn wrapper — which had instructed builders to read a +deleted role path, a live #1011-class bug); eight compressed scar rules with +line-exact enforcement; a machine-readable ownership map whose completeness +contract makes new cross-surface duplication fail CI; dedup via shared +partials (served prompts are per-agent delivery — a prose reference would +delete content from agents' context); and governance sync. + +**N1, honestly: 21,856 → 20,324 served always-on words = −7.0%** against an +aspirational ≥20%. The repair *added* ~207 served words the prompts were +always supposed to have; D3's promotion of three user-global scar rules added +~120; the remaining surface is single-owned prose whose largest block +(protocol.md, 3,703 words) was protocol semantics — a spec Non-goal. That +decomposition is exactly why the successor pivots to shrinking the surface +itself before rebuilding enforcement. + +## Architecture Updates + +The harvest changes no architecture: it syncs shadow-copy *content* to what +the skeleton already ships and lands record-keeping artifacts. The +resolver, the shadow tree's existence, and all governance docs remain as on +main. (The branch's arch-doc rewrites — C6 retirement, single-owner +invariant, glossary corrections — apply to a world where the deletion merged; +they stay with the branch and inform the successor.) + +One architectural *fact* the audit established that main's docs do not yet +state: `copyProtocols`/`copyRoles` in `scaffold.ts` are vestigial (uncalled by +init/adopt/update), so fresh adopters have no shadow tree — this repo's was a +historical artifact. Recorded here for the successor rather than edited into +arch.md out-of-band. + +## Lessons Learned Updates + +Not routed into the shared lessons docs (those edits stayed with the branch); +recorded here for the successor and for MAINTAIN to promote as it sees fit: + +- **A detector that reports without failing is a detector nobody reads.** + #1210 saw this drift for months; Spec 746's comments described it + ("PRE-EXISTING and not Phase 1's responsibility") before stepping around + it. Detection was never the gap — consequence was. +- **Frozen-file baselines are drift bombs.** Spec 746's pure-addition + baselines froze then-drifted files; the legitimate reconciliation broke + them twice (branch and harvest). Derive baselines from current content + minus the asserted insertion instead of freezing snapshots. +- **The corpus is live — a baseline can measure itself.** This project's own + review verdicts contaminated its behavioural baseline (160 → 163) and its + thread file contributed a scar-mining hit; self-exclusion + (`SELF_PROJECT_DIR` / `SELF_FILE_PREFIXES`) had to be built in. +- **"The data exists" ≠ "the metric resolves."** B2 as first specified + ("rounds to unanimous approve") could never resolve: 0 of 48 terminal plan + phases end unanimously — porch advances on rebuttal. +- **Served surfaces dedup by include, not by reference** — each agent sees + only its own prompt; and the measurement must count expanded words or the + metric rewards moving text between files. +- **Sweep-scope failures dominated review iterations**: fixing the flagged + instance instead of the class, grepping for a rule's wording instead of the + act's, repointing a path while preserving the fetch instruction. +- **Gate-rejection counts are not minable** from committed history (no + `rejected` state exists; `requested_at` is overwritten) — a porch + gate-event append-log would fix this (MAINTAIN candidate). + +## Deviations from the Plan + +The implementation deviations (metrics-script location, B2 redefinition, +escalation-deadline correction, reference→include model, T13(b) +rescheduling) are recorded in the phase rebuttals on the branch. The terminal +deviation is this outcome itself: plan phases 4–8's deliverables were built, +reviewed, and then deliberately left unmerged by the pr-gate decision. + +## Flaky Tests + +None skipped, in either the branch suite (3,744 green at close) or the +harvest suite. + +## Verify Phase + +The original verify plan (T13(b) spawn probe + M12b behavioural comparison +with rollback triggers) applied to the unmerged machinery and does not run. +The behavioural baseline remains valid as the **pre-successor** measurement: +the successor project should treat `1252-behavior-baseline.md` (B1 51.88%, +n=160, self-excluded) as its "before" and re-run +`packages/codev/scripts/measure-prompt-behavior.ts` after its rewrite lands. + +## Follow-ups + +- **Successor project** (charter held by Waleed): radical surface shrink + (blog-principles judgment rewrite, >50% target, frontier-models-only), + then fit-for-purpose enforcement; **carries forward the scar-registry + concept and the eight D3-ratified rule wordings**, the ownership-map + design, and this project's audit + baselines. +- #1276 (multi-model tiering) and #1277 (controlled A/B eval) — filed during + Phase 8; both now feed the successor rather than this branch. +- Reference branch: `builder/spir-1252` (PR #1278, closed unmerged) — the + complete machinery with its review history, preserved until cleanup is + ordered. diff --git a/codev/roles/architect.md b/codev/roles/architect.md index aa07e6fbd..56cac231d 100644 --- a/codev/roles/architect.md +++ b/codev/roles/architect.md @@ -29,7 +29,7 @@ Builders work autonomously in isolated git worktrees. The Architect: 1. **`git status`** — Ensure worktree is clean (no uncommitted changes) 2. **Commit if needed** — Builders branch from HEAD; uncommitted specs/plans are invisible -3. **`afx spawn N --protocol `** — `--protocol` is **REQUIRED** (spir, bugfix, tick, etc.) +3. **`afx spawn N --protocol `** — `--protocol` is **REQUIRED** (spir, aspir, air, bugfix, etc.) The spawn command will refuse if the worktree is dirty (override with `--force`, but your builder won't see uncommitted files). @@ -244,13 +244,14 @@ afx cleanup -p 0042 2. **DO NOT commit directly to main** - All changes go through builder PRs 3. **DO NOT use `afx send` for long messages** - Use GitHub PR comments instead 4. **DO NOT run `afx` commands from inside a builder worktree** - All `afx` commands must be run from the repository root on `main`. Spawning from a worktree nests builders inside it, breaking everything. +5. **DO NOT `cd` into a builder worktree** - All CLI tools (`afx`, `porch`, `consult`, `codev`) are global commands that work from any directory. If a command fails, debug it — don't cd into the worktree. Use absolute paths with the Read tool to inspect builder files (e.g., `Read /path/to/.builders/0042/codev/specs/...`). ### ALWAYS Do These: 1. **Create GitHub Issues first** - Track projects as issues before spawning 2. **Review artifacts before approving gates** - (Strict mode) Read the spec/plan carefully 3. **Use PR comments for feedback** - Not terminal send-keys 4. **Let builders own their work** - Guide, don't take over -5. **Stay on `main` at the repo root** - All architect operations happen from the main workspace +5. **Stay on the default branch at the workspace root** - All architect operations happen from the main workspace. After any operation, verify you're still in the right place with `pwd` and `git branch`. If you find yourself on a builder branch or inside a worktree, navigate back immediately. ## Project Tracking @@ -267,28 +268,30 @@ gh issue view 42 Update status as projects progress: - `conceived` → `specified` → `planned` → `implementing` → `committed` → `integrated` -## Working with Area Labels +## Working with Project Labels + +If your project uses prefix-structured labels (e.g. `area/*`, `team/*`, `priority/*`) to organize issues, the recipes below are the architect-specific bulk operations — substitute `` and `` for your project's actual labels. (Skip this section if your project doesn't use prefix-structured labels.) **Operational recipes:** ```bash # Confirm the current label vocabulary (use before any label op to catch drift) -gh label list --search area/ +gh label list --search "/" -# Group: tally open issues by area +# Group: tally open issues by /* label gh issue list --state open --limit 500 --json number,title,labels --jq \ - 'group_by([.labels[].name | select(startswith("area/"))]) | .[] | "\(.[0].labels[] | select(.name | startswith("area/")).name): \(length)"' + 'group_by([.labels[].name | select(startswith("/"))]) | .[] | "\(.[0].labels[] | select(.name | startswith("/")).name): \(length)"' -# Edit: change area on a single issue -gh issue edit --remove-label area/old --add-label area/new +# Edit: change a label on a single issue +gh issue edit --remove-label / --add-label / -# Audit: find open issues with no area label +# Audit: find open issues with no /* label gh issue list --state open --limit 500 --json number,title,labels \ - --jq '.[] | select([.labels[].name] | any(startswith("area/")) | not) | "#\(.number) \(.title)"' + --jq '.[] | select([.labels[].name] | any(startswith("/")) | not) | "#\(.number) \(.title)"' -# Bulk-move: relabel all open `area/` issues to `area/` -for n in $(gh issue list --state open --limit 500 --label area/old --json number --jq '.[].number'); do - gh issue edit "$n" --remove-label area/old --add-label area/new +# Bulk-move: relabel all open / issues to / +for n in $(gh issue list --state open --limit 500 --label / --json number --jq '.[].number'); do + gh issue edit "$n" --remove-label / --add-label / done ``` diff --git a/codev/state/spir-1252_thread.md b/codev/state/spir-1252_thread.md new file mode 100644 index 000000000..59f2f1017 --- /dev/null +++ b/codev/state/spir-1252_thread.md @@ -0,0 +1,826 @@ +# spir-1252 — Prompt architecture: single-owner rule + +## Specify phase — measurement first + +Before drafting, I measured the actual prompt surface rather than trusting the +issue's "~45k words / 4–6 duplicates" estimate. Two findings changed the shape +of the spec. + +### Finding 1: the real duplication axis is the `codev/` ↔ `codev-skeleton/` shadow tree + +The issue lists CLAUDE.md/AGENTS.md, hot-tier docs, protocols, phase prompts, +roles, skills. It never mentions the largest one: **63 markdown files +(45,183 words) exist in BOTH `codev/protocols/` and `codev-skeleton/protocols/`**, +plus `codev/roles/` ↔ `codev-skeleton/roles/`. + +This is not benign. Tier 2 (`codev/`) *wins* over tier 4 (skeleton) in the +four-tier resolver, so our self-hosted instance runs on the local copies — +and **19 of them have drifted** from the skeleton. + +### Finding 2: the drift is live and it bit this very builder + +`codev/protocols/spir/builder-prompt.md` (636 words, served) is missing the +entire **"Multi-PR Mechanics"** and **"## Verify Phase"** sections that +`codev-skeleton/protocols/spir/builder-prompt.md` (824 words, shipped) has. + +My own spawn prompt has no verify-phase instructions. Adopters get them; we +don't. That is the drift failure mode, demonstrated, not hypothesized. + +`codev/protocols/spir/protocol.md` differs from the skeleton by 138 diff lines. +Last touched by the same merge commit (d49f5f47, 2026-06-13) — they were forked +and left to rot. + +### Finding 3: duplication count is worse than the issue estimated + +The `git add -A` prohibition appears in **18 logical locations** (36 files +counting the shadow mirror), in **9 distinct wordings** — from +"Never `git add -A` / `.` / `--all` — stage files explicitly" to a full +"🚨 ABSOLUTE PROHIBITION 🚨" block. Not 4–6; and the wording variance is the +ambiguity cost the issue predicts. + +### Measured surface (words) + +| Surface | Words | Files | Load | +|---|---|---|---| +| CLAUDE.md | 5,773 | 1 | every session, auto | +| AGENTS.md | 5,773 | 1 | byte-identical twin | +| arch-critical + lessons-critical (HOT) | 736 | 2 | in CLAUDE.md *and* every phase prompt | +| arch.md + lessons-learned.md (COLD) | 41,510 | 2 | on demand | +| codev-skeleton (all) | 77,956 | 113 | resolver-dependent | +| codev/protocols (shadow) | 47,511 | 66 | wins over skeleton | +| .claude/skills | 16,743 | 14 | on demand | +| my actual spawn prompt | 4,891 | — | once per builder | +| my actual porch phase task | 1,395 | — | per phase iteration | + +### Consequence for the spec + +The issue framed this as a compression problem ("token cost"). The measurement +says the *drift* problem is bigger and more urgent than the token problem, and +that the shadow tree is where both concentrate. I am specifying accordingly: +ownership map + shadow-tree elimination first, compression second. Deduplication +of scar rules stays off the table per the issue. + +Existing enforcement machinery to build on (not reinvent): `governance-sweep`, +`hot-tier`, `skeleton`, `protocol-prompt-audit`, `framework-ref-audit`, +`skill-parity` test suites. + +## Specify iter-1 — 3-way consultation + +Gemini APPROVE / Codex REQUEST_CHANGES / Claude COMMENT. Codex and Claude each +found real problems. I accepted all nine issues; two of them corrected factual +errors of mine. + +### The one that mattered most (Claude) + +I claimed "no test asserts that codev/protocols/ matches +codev-skeleton/protocols/." **That was wrong.** +`packages/codev/src/lib/protocol-drift-audit.ts` (#1210) already exists — +`auditProtocolDrift()`, `FRAMEWORK_DRIFT_DIRS = ['protocols','consult-types', +'roles']`, classifying `identical | differs` — and `codev doctor` already calls +it at `doctor.ts:947`. + +So drift has been *detected and reported all along*. Our 17 drifted files were +visible to `codev doctor` and ignored. The gap is not detection; it is that +nothing **fails the build**. That narrows M2 enormously: wire the existing +audit into a CI gate, don't write a detector. + +### Count corrections (Claude) + +My "19 drifted" conflated drifted files with local-only ones. Actual: **17** +(16 protocols + 1 roles). Separately there are **3 local-only** entries — +`release/` and two `maintain/templates/` files — which must survive deletion. +Added test T8 for that. Clarified 63 `.md` / 73 `md+json` / 76 including roles. + +### Structural fixes (Codex) + +Codex was right that the spec's end state depended on unanswered critical +questions, and that M2 permitted two incompatible outcomes. Restructured: +M1–M7 are required and Q1-independent; M8–M10 (deletion) are conditional on +architect approval. Added Appendix A (ownership-map schema + what an +"instruction class" is), Appendix B (compatibility audit), Appendix C (tiering +selector/fallback/scope). + +### What I found while verifying Codex's feasibility concern + +Codex claimed many literal `codev/protocols/` references exist. True — 26 +non-test, 95 in tests. But checking them changed the picture twice: + +1. **`scaffold.ts` exports `copyProtocols`/`copyRoles`** which copy the skeleton + into `codev/`. I briefly thought this meant every adopter has a shadow tree. + Then I grepped for callers: **neither is called by init, adopt, or update.** + They are dead code. So adopters have NO shadow tree — ours is a historical + artifact from when scaffolding did copy. That *strengthens* the deletion + case (deleting makes us match adopters) and adds M9: remove the dead + functions before someone rewires them. +2. **Production consumers route through the resolver** (`consult/index.ts:175` + → `readCodevFile`; `porch/protocol.ts` → `resolveCodevFile`). The rest are + comments and error strings. No direct-read consumer found — deletion is + lower-risk than iter-1 judged. + +A1 rewritten accordingly: iter-1 asserted "unintentional fork" without +establishing why; now it is evidence-backed. + +### Still architect-owned + +Q1 (may the shadow trees be deleted) and Q3 (is the scar list complete) are not +mine to answer. The spec no longer *depends* on Q1 — a "no" simply drops +M8–M10 and still ships B + C. + +## Held at spec-approval gate — awaiting full architect decision set + +Architect directed an iter-2 review, then countermanded ~40s later: amend the +spec with the *complete* decision set first, then run a **single** iter-2 +review against the fully-amended spec. Sensible — reviewing a spec that shifts +underneath the reviewers wastes a round. + +State when the countermand landed: I had made the Q2 amendment but had **not** +launched any consult. Nothing to abort. + +**D1 recorded (answers Q2)**: skeleton is authoritative for all 17 drifted +files, reconciled file-by-file — classify each divergence as *missing content* +(take skeleton) or *possible deliberate local content* (**escalate, never +silently overwrite**). Governs M3. Explicitly does NOT decide Q1: reconciling +the files and deleting the tree are separate acts, and D1 holds either way. + +**Frozen pending answers**: Q1 (shadow-tree deletion → M8–M10) and Q3 (scar +registry contents → M5) and Q4 (tiering selector). Added a "Pending — do not +act" clause under Architect Decisions so the constraint is legible to anyone +reading the spec, not just to me. + +Making no further spec edits until the full decision set arrives. + +## Specify iter-2 — decisions D1–D4 absorbed, second 3-way review + +Architect relayed the full decision set: D1 skeleton-authoritative, D2 delete +approved (+ local-unique safeguard), D3 eight scar rules compressed, D4 tiering +cut. Amended spec (442cc08a), then ran iter-2. + +**Verdicts: Gemini APPROVE / Codex REQUEST_CHANGES / Claude APPROVE.** +6 points, all accepted. Iter-1 drew 9 issues; iter-2 drew 6, only 2 blocking — +and both were "your enforcement has a hole," not "your analysis is wrong." +Codex and Claude each independently re-verified the empirical basis; it held. + +### The hole worth remembering (Codex CX-1) + +M4/T7 iterate over the ownership map's entries and check each has one owner. +**A map listing 3 of 40 instruction classes passes cleanly.** Tests green, +artifact looks rigorous, single-owner rule covers a fraction of the surface, +nothing signals the gap. + +I built the exact disease this spec attacks — enforcement that measures only +what you already told it about — into the fix for it. Fixed with a declared +inventory boundary + mechanical candidate extraction + mandatory +mapped/scar/out-of-scope disposition + T12 failing on anything undispositioned. + +Added unprompted: T12 must be validated against a *seeded* normative line. A +completeness test over an empty candidate set passes vacuously and looks +identical to a healthy one. Having just been caught by one vacuous-pass hole, +leaving another in would be careless. + +### CX-2 — escalation is a transition, not a destination + +M11 said "escalate to the architect" and stopped. No terminal state, so the +criterion couldn't be judged complete or incomplete. Added TS1–TS4, made +"pending escalation" explicitly non-terminal, completion = all 76 in TS1–TS4 +with zero open escalations. + +Judgement call flagged to the architect: unresolved escalation converts to TS3 +(keep local, documented, allowlisted) + follow-up issue, rather than +hard-blocking. Trades a little residual shadow tree for a guarantee the project +can't stall on an unanswered question. Also marked TS2 (promote into skeleton) +as *preferred* over TS3 — TS3 knowingly re-creates a shadow copy, and if it +became the default the spec would quietly rebuild what it removed. + +### Smaller but real (Claude CL-2) + +Split T13 into automated CI assembly-check (a) + one manual real spawn (b). +Keeping (b) matters: (a) can pass while the live spawn path reads different +files — which is *exactly* how this project's own prompt lost its Verify Phase +section. An ambiguous "inspect the prompt" would have collapsed into (a) alone. + +Gemini independently endorsed the M11 → M3 → M8 sequencing and the M5-before-C +scar chain — both builder-originated, so worth recording that they survived +outside review. + +No open questions remain. Back to the spec-approval gate. + +## Plan phase — 8 phases drafted + +Spec approved (human approval relayed via architect; I ran `porch approve` per +the workspace convention that the builder executes it, architect never does). + +**One ordering decision beyond what the spec fixed.** The spec pins +`M11 → M3 → M8` and "M5 green before Approach C," but says nothing about where +scar compression sits relative to deletion. Compressing eight rules across ~36 +files when half get deleted two phases later is wasted work — and worse, risks +reconciling a compression edit against a skeleton that never got it. So +**Phase 5 (scar compression) runs AFTER Phase 4 (deletion)**, still well before +Phase 7 (dedup). Satisfies the real constraint; avoids the trap. + +Phases: 1 drift gate + baseline · 2 local-unique audit · 3 reconcile · +4 compat audit + removal · 5 scar registry · 6 ownership map · 7 dedup + +measure · 8 governance sync + E2E. + +Four of eight phases land before a single duplicated word is removed. That's +deliberate — the spec's finding is that drift, not token count, is urgent, and +D2 requires nothing codev-specific be lost. + +Notes to self for implementation: +- Phase 1's allowlist starts POPULATED with the 17 drifts (justified "pending + Phase 3"). A gate that fails on commit is a gate someone disables. +- Phase 5's real risk is meaning loss, not brevity. Each compressed wording must + retain prohibition + scope + any escape hatch (rule 2's "use --resume, and ask + when in doubt"). Dropping the escape hatch turns guidance into a dead end. +- Phase 7: report the N1 figure honestly even if under 20%. Likely shortfall is + structural — most always-on words are CLAUDE.md prose that's already + single-owned, not duplicated rules. Do NOT strip content to hit a number. +- Phase 8 must file the D4 tiering follow-up issue. + +## Plan iter-1 — 3-way review + +**Gemini APPROVE / Codex REQUEST_CHANGES / Claude APPROVE.** 6 points, all +accepted. None changed phase structure, ordering, or scope — all six were +"specify this existing phase more precisely." + +### Codex CX-1: I implemented half of my own criterion + +M10 says resolver equivalence AND byte-identical assembled prompt. Phase 4 +planned only the former — and the gap was *disguised*, because Phase 8's T13 +looks like it covers assembled prompts. A reader would reasonably conclude M10 +was satisfied across two phases when neither asserted byte-identity. + +Why it matters: per-file resolution can be correct while assembly still differs +(template ordering, {{project_id}} interpolation, a fragment from a different +tier). Phase 4's entire claim is "deletion is a no-op for what agents receive" — +only byte-identity shows that. Fixed: snapshot each protocol's prompt +pre-deletion, assert byte-identical after. + +Also wrote down that 4d(ii) and T13 have *opposite* expectations — 4d(ii) says +nothing changed; T13 says the right content is present after compression/dedup +deliberately changed things. Easy for a later reader to collapse them. + +### Codex CX-3: the drift disease, in miniature, in my own plan + +I wrote "remove copyProtocols/copyRoles plus their scaffold.test.ts cases." +But only copyRoles has tests — and **I had established that myself** during the +spec phase and written it in the iter-1 spec rebuttal. Verified once, +paraphrased from memory later, drifted in the restatement. + +That's exactly what this project is about. Worth remembering: my errors don't +cluster in analysis, they cluster in *restating things I already verified*. + +### Claude CL-1: better than "minor" + +T11 parses the Phase 2 audit doc, but I described the doc loosely. A builder +writing free-form prose would make T11 unimplementable — and the natural fix +under pressure is to weaken T11, which would quietly remove the guarantee that +nothing was deleted unaudited. That's the most important safeguard in the plan. +Now specified as a fixed 5-column table with enumerated values. + +### CX-2: phase boundary contradiction + +Phase 5 said "replace on every surface"; Phase 8 said "apply Phase 5's wordings +to CLAUDE.md/AGENTS.md." Two readings, one dangerous: Phase 5 skips the two +most-read surfaces and still claims M5 green — leaving a hole exactly where the +most-read scar rules live, right before Phase 7 starts stripping text. Phase 5 +now owns all scar edits; Phase 8 only parity-checks N3. + +Gemini again endorsed the Phase 5-after-Phase 4 placement (my discretionary +call) and the M11 → M3 → M8 ordering. + +## Amendment D5 — behavioural-impact measurement (M12) + +Architect surfaced a real gap I'd missed: the issue's guidance made impact +measurement mandatory ("a trim proposal without a way to evaluate it is not +accepted"), and my M6 word-counts + structural tests measure the PROXY, not the +effect. My safeguards were asymmetric — M5/T6 protect against a scar rule being +deleted or reworded; nothing detected "rule still present, compliance dropped." + +### Research first — two requested metrics turned out unminable + +Before proposing anything I checked what the repo actually stores. Worth +recording because the answer shaped the whole design: + +- `codev/projects/*/*.txt` (raw consult logs) are **GITIGNORED** (.gitignore:59). + No historical consult output survives. Biggest constraint. +- **Gate rejection counts: NOT MINABLE.** Across all 201 projects gate `status` + only ever takes approved|complete|in_progress|pending — there is no `rejected` + state, and `requested_at` is a scalar that a re-request overwrites. A + rejected-then-approved gate is indistinguishable from a clean one. Said so in + Appendix D §2 rather than quietly dropping it. +- **Tokens/phase: PROSPECTIVE ONLY.** `consult stats` is a rolling 30-day local + DB (3239 invocations, $1426). No Feb–Jun history → forward snapshot only. +- What IS minable: `history[].reviews[].verdict` — but only **17 SPIR projects** + have non-empty history (populated for SPIR's per-plan-phase loops, empty for + pir/bugfix/air). Plus 211 reviews + 139 threads for keyword mining. + +So: B1 REQUEST_CHANGES rate, B2 rounds-to-approve, B3 scar-violation mining, +B4 phase iterations, B5 forward cost snapshot. Sample n=17. N=10 verify window +with ≥3 SPIR (the SPIR minimum is the binding constraint, not the total). + +### Things I made sure to state rather than paper over + +- **The add/remove confound**: this project both restores content (M3) and + removes it (D3 compression, S1 dedup). A null result could be two real effects + cancelling. Mitigated by attributing rollback to specific commits, not the + project as a whole. +- **Rollback targets trims, never repairs.** Phases 1–4 restore correct content; + reverting them reintroduces the drift bug. Only Phases 5 and 7 are candidates. +- **Inconclusive is a real outcome.** <10 projects or <3 SPIR ⇒ do NOT declare + success. Absence of data is not a no-regression result. +- **B3 is the metric that matters most** and is the fuzziest — it's the one that + would catch a compressed scar rule losing force. Script must emit excerpts, + not just counts, so a human adjudicates. +- Honest ceiling: n=17 vs N=10 detects a LARGE regression. The strongest claim + available is "no evidence of harm at this sample size," not "proved + beneficial." That's why M12c defers a real A/B — and I flagged the deferral as + a genuine weakness, not a formality. + +Baseline must land in Phase 1 (step 1b) — Phases 3/5/7 all alter served content, +so any later capture has no clean "before." + +## D5 delta review — Gemini APPROVE / Codex REQUEST_CHANGES / Claude APPROVE + +Codex found two real defects. Gemini and Claude both approved. **The majority +was not the signal.** + +The split is instructive: Gemini and Claude verified the *data-availability* +claims (correct — all three confirmed gate-rejections unminable, consult +prospective-only, 17 SPIR sample). Codex went further and checked whether the +*metric definitions actually resolve against that data*. That's where both +defects were. + +### CX-1: B2 was unmeasurable and would have failed silently + +"Rounds to unanimous approve" — but **0 of 48 terminal plan phases end with +3× APPROVE**. I re-derived it rather than trusting the claim: + + 20 (APPROVE, APPROVE, REQUEST_CHANGES) + 12 (APPROVE, REQUEST_CHANGES, REQUEST_CHANGES) + 7 (REQUEST_CHANGES × 3) + +**Porch advances a phase on builder rebuttal, not consensus.** So B2 would never +resolve — and would have looked fine: script runs, emits a number, number is +meaningless. + +The lesson worth keeping: "the data exists" ≠ "the metric resolves." I verified +the former meticulously (all 3 reviewers confirmed Appendix D §1–2) and never +checked the latter for my own definition. + +Redefined B2 = max(iteration) per plan_phase. Then computed real baselines: +**B1 = 51.9% REQUEST_CHANGES** (n=160), B2 mean 1.12 (n=49), B4 mean 3.06. + +That surfaced something else: **B2's range is 1–2, mean 1.12 — nearly no +variance**, so it can't detect a subtle regression. Demoted B2/B4 to advisory, +named B1 load-bearing, made the soft threshold concrete (>25% on 51.9% = above +~64.9%). + +### CX-2: T14 contradicted itself + +T14 demanded "same commit ⇒ same B1–B5" while B5 comes from a rolling 30-day +machine-local DB. Both halves written by me in the same amendment. Scoped +determinism to B1–B4; B5 now explicitly advisory/non-deterministic, drives no +trigger. Kept rather than dropped — cost/duration is useful context for +interpreting a B1 move; the failure was pretending it was reproducible. + +### Also worth recording + +Gemini endorsed the n=1 hard trigger on B3, reasoning scar rules exist BECAUSE +the catastrophe already happened once — so one verified recurrence justifies +reverting compression. Good framing; a single-incident trigger looks aggressive +without it. + +Process note: my first 3 delta consults failed — `--prompt` and `--type` are +mutually exclusive. Should have checked the consult skill first. Re-ran in +general mode with a scoped brief file. No artifacts affected. + +## Phase 1 COMPLETE — drift gate + both baselines + +Plan approved (human, relayed). Phase 1 delivers M2, M6, M12a. + +**Drift gate** (`shadow-drift-gate.test.ts`): calls the existing +`auditProtocolDrift()` — did NOT reimplement detection. 4 tests + 1 skipped +(Phase-3 lifecycle guard, flips on in Phase 3). + +Allowlist seeded with exactly the 17 drifted files, each justified +`PENDING_RECONCILE → Phase 3 (D1)`. Two guards make it self-limiting: every +entry needs a non-empty note, and a *stale-entry* test fails if an allowlisted +file stops drifting. So the allowlist can't outlive its purpose. + +The BITES test matters most: seeds a real divergence in a temp workspace and +requires detection. A gate that has only seen a clean tree is indistinguishable +from a no-op — which is precisely how #1210 sat unread for months. + +**Word baseline**: ALWAYS_ON = **24,614** (CLAUDE.md 5,773 + spawn 4,891 + +1,395×10 phase tasks). Matches the spec's ~24,600 estimate. Deliberately +excludes AGENTS.md (byte-identical twin — one loads per session, not both) and +the hot-tier files (already inlined in CLAUDE.md's count). Double-counting +either would inflate the baseline and flatter the Phase 7 result. + +**Behavioural baseline** (M12a): B1 **51.88%** REQUEST_CHANGES (n=160 verdicts), +B2 mean 1.12 (n=49 phases), B4 mean 3.06 (n=18 projects). Reproduces the figures +published in spec Appendix D. + +B3: 45 candidate hits across 350 files — and inspection confirms they're mostly +false positives (discussions *about* auto-approve, not violations). Exactly why +the script emits excerpts, not counts. A bare "45 violations" would have been +actively misleading. + +### Deviation from plan: script location + +Plan said `scripts/measure-prompt-behavior.ts` at repo root. It needs `js-yaml`, +which a root script **cannot resolve** in this pnpm workspace (root node_modules +lacks it). Moved logic to `packages/codev/src/lib/prompt-behavior-metrics.ts` +beside `protocol-drift-audit.ts`/`framework-ref-audit.ts` — same module shape, +unit-testable, deps resolve — with a thin CLI at +`packages/codev/scripts/measure-prompt-behavior.ts`. Spec + plan updated to the +real paths. The bash surface script stays at root (no deps). + +### Test-suite investigation worth recording + +First full run: 49 files / 62 tests failing. Easy to write off as pre-existing — +but I verified instead. All were `@cluesmith/codev-core` import errors: the +workspace wasn't built. Building core fixed 48 files. The last 8 were +"integration with real shellper" tests needing `dist/terminal/shellper-main.js`; +`npx tsc` produced it and all 90 passed. + +**Full suite now 193 files / 3772 tests green, 0 failures.** No flaky tests to +skip. Lesson reinforced: "pre-existing failure" is a claim requiring evidence, +not a default assumption — a fresh worktree just isn't built. + +## Phase 1 iter-1 review — Gemini/Claude APPROVE, Codex REQUEST_CHANGES ×2, both fixed + +Codex again the one checking whether the numbers actually mean anything: + +1. **Hardcoded spawn/phase word counts** (4891/1395) would have silently + corrupted the Phase 7 N1 delta — reruns would report pre-trim numbers for + the two largest components. Now derived from resolved artifacts; + sensitivity-verified (shrink protocol.md → ALWAYS_ON drops 21,856→18,048). + Reproducible baseline is **21,856** (old 24,614 counted per-project variable + content that no trim can touch). Spec N1 note updated. +2. **B5 never actually captured** — plan said B1–B5, I wrote "capture + separately" and didn't. Snapshot now in the baseline artifact, capture-dated, + marked non-deterministic, drives no trigger. + +Word baseline output now committed too (1252-word-baseline.md). + +## Phase 2 prep findings (read-only, ahead of the audit) + +Diffed all 17 drifted files against the installed skeleton. Early +classifications for the audit: + +- **spir/protocol.md's 85 "local additions" are ROT**, not local-unique: the + old "Porch Orchestration" section uses obsolete underscore gate IDs + (`spec_approval`; porch uses `spec-approval`) and skeleton restructured to + the build-verify model. Local kept what skeleton deleted. +- **roles/architect.md area-label section is ROT with a twist**: skeleton + GENERALIZED it (`/` placeholders for adopters); the codev-specific + `area/` version's content is already owned by CLAUDE.md. Skeleton wins, + nothing lost. +- **Genuine escalation candidates**: + - TICK "Amendments" sections in spir+aspir spec/plan templates — local-only; + no tick protocol dir exists anywhere, but porch's phase prompts still + reference TICK amendments. Ambiguous → escalate. + - `max_iterations: 8` (local) vs `3` (skeleton) in spir+aspir protocol.json — + a real behavioral difference in CMAP loop bounds. Ambiguous → escalate. + +## Iter-2 hiccup worth remembering: the baseline measured itself + +Porch's checks failed on iter-2: the reproduction test expected 160 verdicts, +got 163. Cause: **this project's own iter-1 review verdicts landed in its +status.yaml — the corpus is live and the measuring project was in it.** Same +for B3: my own thread file (which discusses scar rules constantly) was in the +prose scan and had already contributed a hit to the committed baseline. + +Fix: self-exclusion by default (SELF_PROJECT_DIR, SELF_FILE_PREFIXES). +Baseline = pre-project state, so the measuring project's artifacts are out of +scope by definition. Baseline artifact regenerated clean: 160 verdicts, 349 +files, 44 hits. New test pins the exclusion. + +Observer effect in the smallest possible lab. The verify phase needs to +remember this too: when comparing, 1252's own artifacts stay excluded. + +## Phase 1 iter-2: unanimous APPROVE. Phase 2 (M11 audit) executed. + +Audit committed: codev/resources/1252-shadow-tree-audit.md — **77 rows** (spec +said 76; mechanical enumeration caught consult-types/integration-review.md, +which my hand count missed). 60 identical→TS1, 13 drifted-rot→TS1, 4 +local-unique→pending, escalated to architect: + +1. **max_iterations 8 vs 3** (spir+aspir json) — CMAP loop bound, no config + override exists for it. TS1/TS2/TS3 options laid out. +2. **cwd: packages/codev** (air+bugfix json) — real monorepo functionality; + recommended migrating to .codev/config.json porch.checks (Spec #550 + sanctioned mechanism) then TS1. Noted spir/aspir only pass today because + the repo root happens to have build/test scripts. + +Key evidence trail in the audit: TICK retired (all local TICK refs = rot, +though skeleton's own porch prompts still emit TICK-amendment language — +flagged for Phase 8); obsolete underscore gate IDs prove spir/protocol.md's +"additions" are pre-restructure leftovers; architect.md's area/ recipes are +owned by CLAUDE.md so the skeleton's generalized version loses nothing. + +T11 parses the audit table (6-column contract, enumerated values): 5 tests +green, completion guard (zero pending) phase-gated like the drift gate's. +Also armed: a guard that fails if any file vanishes while pending/TS3. + +## Phase 2 iter-1: Gemini/Claude APPROVE, Codex REQUEST_CHANGES — accepted, remedy differed + +Codex caught my plan contradicting itself: Phase-2 criteria said "zero rows +pending" while the spec's own sequencing designs for open escalations through +Phase 3 (M3 "skips anything M11 escalated") with the real deadline at Phase 4 +(M8 "gated on M11 ... escalations ruled on"). + +The important part: satisfying the criterion AS WRITTEN would have meant +invoking the TS3 escape hatch on all 4 escalations within the hour of sending +them — before the architect could realistically rule. The escape hatch is for +escalations that CANNOT be resolved, not for skipping the wait. A phase +criterion must never strong-arm a human gate. + +Fix: plan criteria amended (deadline = Phase 4); T11 gains an AUTOMATIC guard — +the moment any audited file is deleted, zero pending rows may remain. Deletion +arms the assertion; no human has to remember a flag. That answers Codex's +deeper point (manual skip-flag = phase passes while violating itself) more +durably than resolving the escalations prematurely would have. + +Also: spec count corrected in place (77 shadow copies; hand count missed +consult-types/integration-review.md) and Claude's cosmetic header arithmetic +fixed (73 protocol files, not 74). + +## Phase 3 — 13 rot files reconciled (M3) + +Skeleton copied over the 13 rot files per D1. The headline drift is repaired: +codev/protocols/spir/builder-prompt.md now carries Multi-PR Mechanics, Verify +Phase, and "Entering verify phase." — the next SPIR builder spawned in this +repo gets its verify instructions. + +Remaining divergence: exactly the 4 escalated json files. Allowlist shrunk +17 → 4, all re-tagged ESCALATED with adjudication citations. PHASE_3_COMPLETE +flipped, so the no-PENDING_RECONCILE guard is now live. Audit rows for the 13 +updated to `identical` with "RECONCILED Phase 3 (was: ...)" notes preserving +the original evidence. + +Still waiting on architect rulings for the 4 (max_iterations, cwd). Phase 4 +cannot start until they resolve — enforced by T11's automatic guard, not by my +memory. + +## Phase 3 addendum: reconciliation broke Spec 746's frozen baselines — instructive + +Porch checks caught baked-decisions.test.ts failing: it froze the THEN-DRIFTED +codev builder-prompts as "pure-addition baselines" (Spec 746), so my +reconciliation's line change (Ready for cleanup. → Entering verify phase.) +violated the pure-addition invariant. First fix attempt (baseline := current) +broke the pollution check — the baseline's true meaning is "file minus 746's +insertion". Rebuilt both baselines as current-minus-Baked-Decisions-block; +all 193 file tests + full suite (3780) green. + +Two durable notes: +1. **M7 catch**: baked-decisions.test.ts reads codev/protocols/* by literal + path — a test-tier consumer that Phase 4's deletion WILL break (ENOENT). + Added to the Phase 4 compat-audit worklist: repoint its codev/ entries at + the skeleton (which after deletion is the single source). +2. Frozen-file baselines are drift bombs: any legitimate later edit fails the + check for the wrong reason. The 746 test's real claim ("the baked section + is a pure insertion") survives only if the baseline is DERIVED + (current minus section), not frozen. + +## Phase 4 — shadow tree DELETED (77 files), M8–M10 complete + +Rulings landed: E1 = TS1 (skeleton's max_iterations 3 — the B2 baseline +evidence decided it), E2 = migrate cwd to .codev/config.json porch.checks for +all four protocols, then TS1. + +E2 carried a verification requirement (config is untracked — does it reach +every porch-check context?). Verified against code: spawn symlinks it into new +worktrees (spawn-worktree.ts:100); CI never runs porch checks; and the decisive +fact — **the migration is defense-in-depth, not load-bearing**, because root +package.json delegates build/test to packages/codev, proven green all session +by spir's cwd-less checks. No context can break. Main-checkout file creation +handed to the architect (can't write outside worktree); fallback covers the +interim. + +Execution: 4 jsons reconciled → 77 shadow copies deleted → roles/ and +consult-types/ gone entirely → 3 local-only files preserved → +copyRoles/copyProtocols removed with a tombstone → M10 manifest (77 sha256s) ++ equivalence test prove deletion was a byte-level no-op for what agents are +served → allowlist EMPTY → all T11 guards green, zero pending. + +**Humbling M7 note**: my step-4a regex classification of test consumers had 4 +false negatives (bugfix-744, governance-sweep, review-prompt-routing, +bugfix-619 — path.join segments / template literals / dirname traversal). The +deletion caught them: ENOENT, loud, at the point of change. 8 repo-readers +total, all repointed at codev-skeleton/. Recorded in the audit: a grep +classification is a hypothesis; the deletion is the test. + +The repo now dogfoods exactly what it ships. C6's mirror-both-trees invariant +is dead for protocols/roles — Phase 8 updates arch-critical accordingly. + +## Phase 4 iter-1: Gemini/Claude APPROVE, Codex REQUEST_CHANGES ×3 — all fixed + +Codex caught me delivering less than my own amended plan: M10(ii) promised +actual assembled-prompt snapshots and I shipped an inference (hashes + purity +argument). The argument was sound but tested the premises, not the conclusion. +Fixed properly: pre-deletion tree reconstructed via git archive, real +buildPromptFromTemplate run against it for all 9 protocols (spir prompt = +30,593 bytes with inlined protocol reference), snapshots committed, live +assembly must reproduce them byte-for-byte. Also added the tier assertion +(resolved path === skeleton path, not just content match). + +Third catch: my mechanical repoint left bugfix-685's parity comparing a file +to itself — a test that can never fail, worse than no test. Removed it AND the +identical defect in baked-decisions (unflagged — fixing the class, not the +instance). Suite count drops 3,767→3,746: vacuous tests deleted, not skipped. + +Gem for the review: Spec 746's comment explicitly described the drift +("skeleton has Multi-PR Workflow / Verify Phase sections that codev/ doesn't +... PRE-EXISTING and not Phase 1's responsibility"). The drift was OBSERVED +and consciously stepped around months before this project. Detection was never +the problem — thirdhand confirmation of the spec's core thesis. + +## Phase 4 iter-3 — Codex found the drift thesis at ANOTHER copy boundary + +Codex's catch was architecturally the best of the project: my repointed tests +validate codev-skeleton/ (source), but resolveCodevFile serves the EMBEDDED +copy at packages/codev/skeleton (build-time cp). Source tests could pass while +agents get stale bytes — the project's own disease at the build boundary. + +Remedy: skeleton-embed-sync.test.ts — bidirectional byte-parity between source +and embedded trees. One guard at the copy boundary instead of rerouting N +tests through the resolver. Source tests keep their commit-surface advantage +(fail where the dev edits, pre-build); the guard ties commit surface to +serving surface. Same single-owner pattern as everywhere else: enforce the +boundary, don't chase consumers. + +Also swept the duplicate-entry residue as a real class this time (after +claiming that once before and missing 742): deduped 744/governance-sweep/ +protocol-prompt-audit AND found three more vacuous pair loops inside +baked-decisions. 3,726 green. + +## Phase 5 — scar registry, eight compressed wordings (M5/D3) + +Registry at codev/resources/scar-rules.yaml; T6 (scar-rules.test.ts) enforces: +count pinned at 8, ids pinned, byte-verbatim presence on every listed surface, +stale-variant sweep, N3 pair check. 28 skeleton files + CLAUDE/AGENTS + +arch-critical + 4 skill copies converged to canonical wordings. + +Wording judgement calls worth recording: +- **human-gates**: NOT "never approve gates yourself" — builders legitimately + run `porch approve` after the human decision is relayed (workspace + convention). Canonical targets the actual sin: treating a gate as approved + WITHOUT an explicit human decision. +- **never-destroy-worktrees** keeps its escape hatch (--resume, then ask) and + its teeth ("what is expendable is never your call") in two sentences. +- Rules 3/7/8 had NO repo surface (lived only in Waleed's user-global + CLAUDE.md); D3's ratification promotes them to repo CLAUDE/AGENTS always-on. + +Fixture fallout handled as explicit acts: prompt snapshots + manifest +regenerated (now pin post-compression state; comment documents the contract), +derived baselines re-derived with the same canonical substitutions. + +CLAUDE.md structure: two 🚨 banner blocks collapsed into topical compact rules ++ new "Scar Rules (always in force)" section housing the five without topical +homes. Net CLAUDE.md shrink measured in Phase 7. + +## Phase 5 convergence saga — 4 iterations, each Codex catch real + +Iter-1: includes() → line-exactness (appended-qualifier attack); -f → -fd; +maintain/protocol.md registered. Iter-2: architect.md self-contradiction; +gate-rule variant on 8 builder-prompts. Iter-3: SECOND afx-anywhere instance +(fixed the flagged line, hadn't swept the file); sweep extended to the class. +Iter-4: "bypass checks" variant ×11 (grepped the rule's phrasings, not the +ACT's); arch-critical exception dissolved by rewriting the now-false two-trees +fact (freed a cap slot; R5 canonical stands line-exact; pulled a sliver of +Phase 8's C6 rewrite forward with justification). + +Meta-lesson accumulating for the review: every iteration's miss was a +sweep-scope failure — fix-the-instance vs fix-the-class, and grep-for-wording +vs grep-for-act. The registry+line-exactness machinery is exactly what makes +these misses eventually impossible instead of eventually repeated. + +## Phase 6 — ownership map + completeness enforcement (M1/M4) + +Data-first (the B2 lesson): built the extractor, ran it over the declared +boundary (26 files), THEN authored the map from what it found. 134 normative +candidates: 23 mapped (7 non-scar duplicated classes), 37 scar +(registry-enforced externally — no double ownership of scar enforcement), +74 file-local singles. + +Design decision that matters: the catch-all disposition would make T12 vacuous +(everything matches /.*/), so the resolver tracks catch-all usage and FAILS +any candidate whose text appears on 2+ files via catch-all. Single-file +process lines pass cheaply; cross-surface duplication — the thing the +single-owner rule is about — always demands an explicit disposition. New +duplication fails CI the moment it's introduced. + +The 7 non-scar classes are enforcement:manual with justifications naming +their Phase-7 flip (a test asserts every manual justification does so). T7's +machinery is fixture-proven (detects restatement on non-owner) and activates +per-class as Phase 7 flips them — same staged-activation pattern as the drift +gate. Two coordination notes recorded in the map: baked-decisions dedup must +move with Spec 746's test suite; pr-single-by-default with bugfix-744's. + +Suite: 3,744 green. + +## Phase 7 — dedup + measurement: the honest N1 story + +**Achieved: 21,856 → 20,314 served always-on words = −7.1% (target was 20%).** +Reporting the shortfall plainly, per the spec's own instruction, with the +structural decomposition: + +- CLAUDE.md −1,421: the real win. Relocated Runnable Worktrees (1,212w of + reference-tier content that existed NOWHERE else — relocation-with-pointer, + not dedup) into agent-farm.md (both trees), trimmed Inter-agent messaging to + summary+pointer (deep grammar owned by agent-farm.md). +- Spawn −169 net: drift REPAIR added ~207 served words (Multi-PR + Verify + sections the prompt was supposed to have!); protocol.md replacement −384. +- Phase task ≈ flat: canonical scar lines ≈ old variants in length. +- D3 compression's CLAUDE.md savings were largely consumed by D3's OWN + promotion of rules 3/7/8 into the repo (+~120) — compressing while adding. + +Why 20% was never reachable in scope: remaining always-on = protocol.md +(3,703w, protocol semantics excluded by Non-goals), porch prompts (process +scaffolding), CLAUDE.md's genuinely-always-on remainder. The 24.6k→20% +arithmetic assumed more of the surface was duplicated rules; most was +single-owned prose. The verify phase will tell us if even −7% moved behavior. + +**Dedup by include, not by reference** — the key design fact: these are SERVED +prompts; a bugfix builder never sees spir's prompt, so prose references would +DELETE content from agents' context. Shared partials +(codev-skeleton/partials/, expanded via resolveCodevIncludes at assembly) give +single AUTHORED ownership with unchanged SERVED bytes. Include support added +to builder-prompt loading (one line, mirroring the protocol_reference path). +5 partials extracted; 6 classes flipped automated; protocol-specific variants +kept via a new `retained_restatements` field (distinct from `references` so +pointer surfaces stay pattern-forbidden — preserving Codex's Phase-6 fix). + +**Measurement integrity catch (self-caught)**: after extraction the proxy +counted AUTHORED builder-prompt words (612) while agents are SERVED the +expanded form (851). Fixed the script to count expanded words — otherwise N1 +would have claimed 239 phantom words. Dedup-by-include changes ownership, not +served bytes; the metric must not reward it. + +Also: extractor probe corrected my map data twice more (soft-mode carriers = +spir/aspir/bugfix/pir/air, NOT experiment; notifications = spir/aspir uniform ++ air/bugfix protocol-specific). Every reference list I wrote from memory was +wrong somewhere; every grep-verified one held. The pattern is now beyond +dispute. + +## Phase 8 — governance sync + E2E disposition + +- CLAUDE/AGENTS: ~15 dangling `codev/protocols/...` prose refs repointed at + codev-skeleton/ (deletion made them point at nothing); RELEASE pointer + annotated as codev-local-by-design. N3 byte-identity held. +- arch.md: Repository Dual Nature rewritten (codev/protocols/ = local-only + protocols exclusively; shadow story + enforcement told in place); new + single-owner invariant added to Invariants & Constraints. +- lessons-learned.md (Critical): "A detector that reports without failing is a + detector nobody reads" + corollaries (frozen baselines are drift bombs; + grep classification is a hypothesis, deletion is the test; measure the + served artifact, not the authored one). +- arch-critical: already restructured in Phase 5 iter-4 (C6 retired there); + caps verified 10 facts / 33 lines. +- Follow-ups filed: #1276 (tiering, D4) and #1277 (A/B eval, M12c). +- **T13(b) rescheduled to VERIFY by architect ruling** — afx spawn has no + branch selector; every spawn branches from main HEAD, so a pre-merge probe + assembles the PRE-change tree and verifies nothing. Discovered constraint, + documented in spec T13 and added to the verify checklist (architect runs the + disposable probe post-merge + post-local-install and records the prompt). + Pre-merge proxy: the nine byte-compared prompt snapshots. + +## Review phase — all 8 phases closed, review doc written, PR next + +Phase 8 closed after 5 rounds (final: unanimous). Full arc: 8 phases, +~20 review rounds, every Codex REQUEST_CHANGES either fixed or rebutted with +evidence that survived subsequent rounds. Review doc at +codev/reviews/1252-prompt-architecture-single-own.md carries the honest N1 +(-7.0%), the verify-phase checklist (T13b probe + M12b comparison), and the +systematic observation that the review loop's recurring value was checking +that enforcement mechanisms BIND, not that artifacts exist. + +## FINAL CHAPTER — harvested, not merged (2026-07-28) + +Waleed's pr-gate decision: PR #1278 does NOT merge. Charter-level re-scope, +not a quality verdict — a successor project will shrink the prompt surface +radically first (blog-principles judgment rewrite, >50% target, +frontier-models-only), then rebuild enforcement fit-for-purpose around the +post-shrink surface. Carrying deletion + registry + map + trims for a surface +about to be rewritten is the wrong risk/reward. + +Harvest executed: a minimal PR off main carrying (1) the 17-file drift +reconciliation — the live bug — with E1 (max_iterations:3) and E2 (cwd via +main's .codev/config.json) applied, plus the 746 baseline re-derivations the +sync forces; (2) the knowledge artifacts: shadow-tree audit, word baselines, +behavioural baseline, measurement tooling + determinism test, the rewritten +review, this thread. + +Stays on this branch (preserved, PR closed unmerged): deletion + equivalence +proofs, scar registry + T6, ownership map + T12, partials + served guard, +CLAUDE.md restructure, all skeleton edits. The scar-registry concept and the +eight ratified wordings move into the successor charter — deferred, not +discarded. + +Worth saying plainly: the measurement work is what made this decision +possible. An honest -7.0% against a 20% aspiration, decomposed to show the +remaining mass was single-owned protocol prose, is exactly the evidence that +says "shrink the surface first, then enforce." The project's most durable +output may be that its numbers argued against its own machinery. diff --git a/packages/codev/scripts/measure-prompt-behavior.ts b/packages/codev/scripts/measure-prompt-behavior.ts new file mode 100644 index 000000000..b820486d9 --- /dev/null +++ b/packages/codev/scripts/measure-prompt-behavior.ts @@ -0,0 +1,19 @@ +#!/usr/bin/env npx tsx +/** + * CLI runner for the behavioural-impact measurement (Spec 1252, M12 / T14). + * + * The logic lives in `../src/lib/prompt-behavior-metrics.ts` so it is + * unit-testable and can resolve this package's `js-yaml` dependency; this file + * is a thin entry point only. + * + * Usage (from packages/codev): + * npx tsx scripts/measure-prompt-behavior.ts + * npx tsx scripts/measure-prompt-behavior.ts --json + * + * Produces the Phase-1 baseline artifact: + * npx tsx scripts/measure-prompt-behavior.ts ../.. \ + * > ../../codev/resources/1252-behavior-baseline.md + */ +import { runCli } from '../src/lib/prompt-behavior-metrics.js'; + +process.stdout.write(runCli(process.argv.slice(2), process.cwd()) + '\n'); diff --git a/packages/codev/src/__tests__/prompt-behavior-metrics.test.ts b/packages/codev/src/__tests__/prompt-behavior-metrics.test.ts new file mode 100644 index 000000000..d422764ee --- /dev/null +++ b/packages/codev/src/__tests__/prompt-behavior-metrics.test.ts @@ -0,0 +1,187 @@ +/** + * Behavioural-metrics baseline script (Spec 1252, M12a / T14). + * + * The metrics exist so that a word-count win cannot be declared while agent + * behaviour silently degrades. For a before/after comparison to mean anything, + * the committed-artifact metrics must be DETERMINISTIC: the same commit must + * yield the same numbers. + * + * B5 (consult cost/duration) is deliberately absent from this module and from + * the determinism assertion — it derives from a rolling 30-day machine-local + * DB, so it is not reproducible from a commit. It is advisory context and + * drives no rollback trigger. (T14 originally asserted determinism over + * "B1–B5", which was self-contradictory; caught at delta review.) + */ + +import { describe, it, expect } from 'vitest'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import { tmpdir } from 'node:os'; +import { measureBehavior, render } from '../lib/prompt-behavior-metrics.js'; + +const REPO_ROOT = path.resolve(__dirname, '../../../..'); + +/** Build a throwaway repo root with a hand-written status.yaml corpus. */ +function fixtureRoot(statusYamls: Record, prose: Record = {}) { + const root = fs.mkdtempSync(path.join(tmpdir(), 'codev-behavior-')); + for (const [proj, body] of Object.entries(statusYamls)) { + const d = path.join(root, 'codev', 'projects', proj); + fs.mkdirSync(d, { recursive: true }); + fs.writeFileSync(path.join(d, 'status.yaml'), body); + } + for (const [rel, body] of Object.entries(prose)) { + const p = path.join(root, rel); + fs.mkdirSync(path.dirname(p), { recursive: true }); + fs.writeFileSync(p, body); + } + return root; +} + +const TWO_PHASE_PROJECT = ` +id: '900' +protocol: spir +history: + - iteration: 1 + plan_phase: phase_1 + reviews: + - {model: gemini, verdict: APPROVE} + - {model: codex, verdict: REQUEST_CHANGES} + - {model: claude, verdict: APPROVE} + - iteration: 2 + plan_phase: phase_1 + reviews: + - {model: gemini, verdict: APPROVE} + - {model: codex, verdict: REQUEST_CHANGES} + - {model: claude, verdict: COMMENT} + - iteration: 1 + plan_phase: phase_2 + reviews: + - {model: gemini, verdict: APPROVE} +`; + +describe('behavioural metrics (M12a / T14)', () => { + it('is DETERMINISTIC over B1–B4 for a fixed corpus', () => { + const root = fixtureRoot({ '900-x': TWO_PHASE_PROJECT }); + try { + const a = measureBehavior(root); + const b = measureBehavior(root); + expect(a).toEqual(b); + // and stable across a re-render + expect(render(a)).toEqual(render(b)); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } + }); + + it('computes B1 as the REQUEST_CHANGES share of all verdicts', () => { + const root = fixtureRoot({ '900-x': TWO_PHASE_PROJECT }); + try { + const m = measureBehavior(root); + // 7 verdicts: 4 APPROVE, 2 REQUEST_CHANGES, 1 COMMENT + expect(m.b1_totalVerdicts).toBe(7); + expect(m.b1_verdictCounts['REQUEST_CHANGES']).toBe(2); + expect(m.b1_requestChangesRate).toBeCloseTo(28.57, 1); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } + }); + + it('computes B2 as max(iteration) per plan_phase, not rounds-to-unanimity', () => { + // The corpus above never reaches 3x APPROVE on phase_1 — which is exactly + // the real-world pattern (0 of 48 terminal phases in this repo end + // unanimously; porch advances on builder rebuttal). A rounds-to-unanimity + // metric would never resolve here; max(iteration) does. + const root = fixtureRoot({ '900-x': TWO_PHASE_PROJECT }); + try { + const m = measureBehavior(root); + expect(m.b2_phaseCount).toBe(2); // phase_1, phase_2 + expect(m.b2_roundsPerPhaseMax).toBe(2); // phase_1 reached iteration 2 + expect(m.b2_roundsPerPhaseMean).toBeCloseTo(1.5, 2); + expect(m.b4_roundsPerProjectMean).toBeCloseTo(3, 2); // 2 + 1 + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } + }); + + it('ignores projects with empty history (pir/bugfix/air contribute nothing)', () => { + const root = fixtureRoot({ + '900-x': TWO_PHASE_PROJECT, + '901-y': "id: '901'\nprotocol: bugfix\nhistory: []\n", + }); + try { + const m = measureBehavior(root); + expect(m.sampleProjects).toEqual(['900-x']); + expect(m.b4_projectCount).toBe(1); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } + }); + + it('survives a malformed status.yaml without aborting the run', () => { + const root = fixtureRoot({ + '900-x': TWO_PHASE_PROJECT, + '902-bad': 'id: [unclosed\n ::: not yaml :::\n', + }); + try { + const m = measureBehavior(root); + expect(m.sampleProjects).toEqual(['900-x']); // bad file skipped, run completes + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } + }); + + it('B3 emits excerpts, not just counts, so a human can adjudicate', () => { + const root = fixtureRoot( + {}, + { + 'codev/reviews/1-x.md': 'I ran git add -A by mistake and it staged secrets\n', + 'codev/state/b_thread.md': 'Reminder: never use git add -A here.\n', + } + ); + try { + const m = measureBehavior(root); + const hit = m.b3_candidateHits.find((h) => h.rule === 'git-add-all'); + expect(hit, 'the confessional line should be flagged').toBeTruthy(); + expect(hit!.excerpt).toContain('git add -A'); + expect(hit!.file).toBe('codev/reviews/1-x.md'); + expect(hit!.line).toBe(1); + + // The prescriptive "never use..." line is documentation, not a breach. + expect(m.b3_candidateHits.filter((h) => h.file.endsWith('b_thread.md'))).toEqual([]); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } + }); + + it('does NOT report gate-rejection counts (the data does not exist)', () => { + // Appendix D §2: no `rejected` gate state is ever persisted and + // `requested_at` is overwritten on re-request, so a rejected-then-approved + // gate is indistinguishable from a clean approval. A plausible-looking zero + // would be worse than an absent metric. + const m = measureBehavior(REPO_ROOT); + expect(Object.keys(m)).not.toContain('gateRejections'); + expect(JSON.stringify(m)).not.toMatch(/gateReject/i); + }); + + it('excludes the measuring project from its own baseline (B1–B4 and B3)', () => { + // The baseline is the PRE-PROJECT state. Project 1252's own status.yaml + // accumulates verdicts with every review iteration, and its thread/review + // files discuss scar rules at length — scanning either makes the baseline + // drift as the project runs. (Discovered live: the reproduction test below + // failed the moment this project's own iter-1 review landed, 160 → 163.) + const m = measureBehavior(REPO_ROOT); + expect(m.sampleProjects).not.toContain('1252-prompt-architecture-single-own'); + for (const h of m.b3_candidateHits) { + expect(h.file).not.toMatch(/spir-1252_|\/1252-/); + } + }); + + it('reproduces the committed baseline numbers on this repo', () => { + // Guards against a refactor silently changing what the baseline means. + const m = measureBehavior(REPO_ROOT); + expect(m.b1_totalVerdicts).toBe(160); + expect(m.b1_requestChangesRate).toBeCloseTo(51.88, 1); + expect(m.b2_roundsPerPhaseMean).toBeCloseTo(1.12, 2); + expect(m.b4_roundsPerProjectMean).toBeCloseTo(3.06, 2); + }); +}); diff --git a/packages/codev/src/agent-farm/__tests__/fixtures/baselines/aspir-builder-prompt.md.baseline b/packages/codev/src/agent-farm/__tests__/fixtures/baselines/aspir-builder-prompt.md.baseline index a50bdd6a8..017d0a2c0 100644 --- a/packages/codev/src/agent-farm/__tests__/fixtures/baselines/aspir-builder-prompt.md.baseline +++ b/packages/codev/src/agent-farm/__tests__/fixtures/baselines/aspir-builder-prompt.md.baseline @@ -52,11 +52,43 @@ Follow the implementation plan at: `{{plan.path}}` {{task_text}} {{/if}} +## PR Strategy + +**Do not autonomously open a PR per implementation phase.** Plan phases ship as git commits within a single PR, not as separate PRs. The plan's instruction that "each phase commits independently" refers to git commits, not PRs. + +By default, the PR is opened during/after the final implement phase, with all phase-commits already on the branch. + +### Architect-requested PRs + +The architect MAY request a PR at any point — for spec review, mid-implementation feedback, slicing a large spec into shippable PRs, etc. When the architect explicitly asks for a PR earlier (or for additional PRs), follow that direction. The prohibition is specifically on the *builder* autonomously deciding to open per-phase PRs without architect request. + +### Multi-PR Mechanics (when the architect requests sequential PRs) + +Your worktree is persistent — it survives across PR merges. When the architect asks for sequential PRs (e.g., to slice a large spec into shippable pieces), use this loop: + +1. Cut a branch, open a PR, wait for merge +2. After merge: `git fetch origin && git checkout -b origin/` — where `` is the branch the architect targets PRs at (usually `main`; check the open PR's `baseRefName` if unsure) +3. Continue to the next slice, open another PR + +**Important**: Do NOT run `git checkout ` — git worktrees cannot check out a branch that's checked out elsewhere. Always branch off `origin/` via fetch. + +Record PRs: `porch done {{project_id}} --pr --branch ` +Record merges: `porch done {{project_id}} --merged ` + +## Verify Phase + +After the final PR merges, the project enters the **verify** phase. You stay alive through verify: +1. Pull the integration branch into your worktree +2. Run `porch done {{project_id}}` to signal verification is ready +3. The architect approves `verify-approval` when satisfied + +If verification is not needed: `porch verify {{project_id}} --skip "reason"` + ## Notifications Always use `afx send architect "..."` to notify the architect at key moments: - **Gate reached**: `afx send architect "Project {{project_id}}: ready for approval"` - **PR ready**: `afx send architect "PR #N ready for review (project {{project_id}})"` -- **PR merged**: `afx send architect "Project {{project_id}} complete. PR merged. Ready for cleanup."` +- **PR merged**: `afx send architect "Project {{project_id}} PR merged. Entering verify phase."` - **Blocked**: `afx send architect "Blocked on project {{project_id}}: [reason]"` ## Handling Flaky Tests @@ -72,3 +104,9 @@ If you encounter **pre-existing flaky tests** (intermittent failures unrelated t 1. Read the protocol document thoroughly 2. Review the spec and plan (if available) 3. Begin implementation following the protocol phases + +--- + +## Protocol Reference (full text) + +{{protocol_reference}} diff --git a/packages/codev/src/agent-farm/__tests__/fixtures/baselines/spir-builder-prompt.md.baseline b/packages/codev/src/agent-farm/__tests__/fixtures/baselines/spir-builder-prompt.md.baseline index f56f68082..2d7f614de 100644 --- a/packages/codev/src/agent-farm/__tests__/fixtures/baselines/spir-builder-prompt.md.baseline +++ b/packages/codev/src/agent-farm/__tests__/fixtures/baselines/spir-builder-prompt.md.baseline @@ -52,11 +52,44 @@ Follow the implementation plan at: `{{plan.path}}` {{task_text}} {{/if}} +## PR Strategy + +**Do not autonomously open a PR per implementation phase.** Plan phases ship as git commits within a single PR, not as separate PRs. The plan's instruction that "each phase commits independently" refers to git commits, not PRs. + +By default, the PR is opened during/after the final implement phase, with all phase-commits already on the branch. + +### Architect-requested PRs + +The architect MAY request a PR at any point — for spec review, mid-implementation feedback, slicing a large spec into shippable PRs, etc. When the architect explicitly asks for a PR earlier (or for additional PRs), follow that direction. The prohibition is specifically on the *builder* autonomously deciding to open per-phase PRs without architect request. + +### Multi-PR Mechanics (when the architect requests sequential PRs) + +Your worktree is persistent — it survives across PR merges. When the architect asks for sequential PRs (e.g., to slice a large spec into shippable pieces), use this loop: + +1. Cut a branch, open a PR, wait for merge +2. After merge: `git fetch origin && git checkout -b origin/` — where `` is the branch the architect targets PRs at (usually `main`; check the open PR's `baseRefName` if unsure) +3. Continue to the next slice, open another PR +4. Repeat + +**Important**: Do NOT run `git checkout ` — git worktrees cannot check out a branch that's checked out elsewhere. Always branch off `origin/` via fetch. + +Record PRs in status.yaml: `porch done {{project_id}} --pr --branch ` +Record merges: `porch done {{project_id}} --merged ` + +## Verify Phase + +After the final PR merges, the project enters the **verify** phase. You stay alive through verify: +1. Pull the integration branch into your worktree +2. Run `porch done {{project_id}}` to signal verification is ready +3. The architect approves `verify-approval` when satisfied + +If verification is not needed: `porch verify {{project_id}} --skip "reason"` + ## Notifications Always use `afx send architect "..."` to notify the architect at key moments: - **Gate reached**: `afx send architect "Project {{project_id}}: ready for approval"` - **PR ready**: `afx send architect "PR #N ready for review (project {{project_id}})"` -- **PR merged**: `afx send architect "Project {{project_id}} complete. PR merged. Ready for cleanup."` +- **PR merged**: `afx send architect "Project {{project_id}} PR merged. Entering verify phase."` - **Blocked**: `afx send architect "Blocked on project {{project_id}}: [reason]"` ## Handling Flaky Tests @@ -72,3 +105,9 @@ If you encounter **pre-existing flaky tests** (intermittent failures unrelated t 1. Read the protocol document thoroughly 2. Review the spec and plan (if available) 3. Begin implementation following the protocol phases + +--- + +## Protocol Reference (full text) + +{{protocol_reference}} diff --git a/packages/codev/src/lib/prompt-behavior-metrics.ts b/packages/codev/src/lib/prompt-behavior-metrics.ts new file mode 100644 index 000000000..e325c440f --- /dev/null +++ b/packages/codev/src/lib/prompt-behavior-metrics.ts @@ -0,0 +1,388 @@ +/** + * prompt-behavior-metrics.ts — behavioural-impact measurement of prompt changes. + * Spec 1252, criterion M12 / Appendix D / test T14. + * + * Lives in `src/lib/` beside `protocol-drift-audit.ts` and + * `framework-ref-audit.ts` — same module shape (a measurement library, unit + * tested), and it needs this package's `js-yaml` dependency, which a root-level + * `scripts/` file cannot resolve in this pnpm workspace. The runnable entry + * point is `packages/codev/scripts/measure-prompt-behavior.ts`. + * + * ## Why this exists + * + * M6 counts words. A word reduction that degrades agent compliance is a loss, + * and word counts cannot see that. This script measures BEHAVIOUR: how often + * reviewers demand changes, how many rounds phases take, and whether scar rules + * are being violated. + * + * Run at Phase 1 (before any prompt content changes) to produce the baseline, + * and again in the post-merge verify phase to compare. + * + * ## What is deliberately NOT measured, and why + * + * Two metrics were requested and cannot be delivered from committed history. + * They are omitted rather than faked, because a plausible-looking zero is worse + * than an absent metric: + * + * - GATE REJECTION COUNTS. Across all projects, a gate's `status` only ever + * takes `approved | complete | in_progress | pending` — there is no + * `rejected` state — and `requested_at` is a scalar that a re-request + * OVERWRITES rather than appends to. A rejected-then-approved gate is + * indistinguishable from a clean first-time approval. (Making this minable + * would need porch to append gate events; out of scope here.) + * + * - HISTORICAL CONSULT TOKENS/COST. Raw consult logs (`codev/projects/*​/*.txt`) + * are gitignored, and `consult stats` is a rolling 30-day machine-local DB. + * There is no historical series to baseline against, so B5 is captured as a + * FORWARD snapshot only — advisory, non-deterministic, and excluded from + * T14's determinism assertion and from every rollback trigger. + * + * Usage (from packages/codev): + * npx tsx scripts/measure-prompt-behavior.ts [repo-root] + * npx tsx scripts/measure-prompt-behavior.ts --json + */ +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as yaml from 'js-yaml'; + +// --------------------------------------------------------------------------- +// Types +// --------------------------------------------------------------------------- + +type Verdict = 'APPROVE' | 'REQUEST_CHANGES' | 'COMMENT' | string; + +interface ReviewRec { + model?: string; + verdict?: Verdict; +} +interface HistoryItem { + iteration?: number; + plan_phase?: string | null; + reviews?: ReviewRec[]; +} +interface StatusYaml { + id?: string; + protocol?: string; + history?: HistoryItem[]; +} + +export interface ScarHit { + rule: string; + file: string; + line: number; + excerpt: string; +} + +export interface BehaviorMetrics { + /** B1 — share of all CMAP verdicts that are REQUEST_CHANGES. */ + b1_requestChangesRate: number; + b1_verdictCounts: Record; + b1_totalVerdicts: number; + /** B2 — review rounds per plan phase: max(iteration) per plan_phase. */ + b2_roundsPerPhaseMean: number; + b2_roundsPerPhaseMedian: number; + b2_roundsPerPhaseMax: number; + b2_phaseCount: number; + /** B3 — candidate scar-rule violations (REQUIRES HUMAN ADJUDICATION). */ + b3_candidateHits: ScarHit[]; + b3_filesScanned: number; + /** B4 — review rounds per project. */ + b4_roundsPerProjectMean: number; + b4_roundsPerProjectMedian: number; + b4_projectCount: number; + /** Sample provenance. */ + sampleProjects: string[]; +} + +// --------------------------------------------------------------------------- +// The eight ratified scar rules (decision D3) and their violation markers. +// +// These patterns look for the DANGEROUS COMMAND / ACT, not for the rule text. +// That is still fuzzy: documentation *about* a rule reads much like a violation +// of it. Hence excerpts, not just counts — a human adjudicates before the hard +// rollback trigger fires. +// --------------------------------------------------------------------------- + +const SCAR_PATTERNS: Array<{ rule: string; re: RegExp }> = [ + { rule: 'git-add-all', re: /\bgit\s+add\s+(-A\b|--all\b|\.(?:\s|$))/ }, + { rule: 'destroy-worktree', re: /\bgit\s+worktree\s+remove\b|\bgit\s+branch\s+-D\b/ }, + { rule: 'destructive-git', re: /\bgit\s+(reset\s+--hard|checkout\s+--\s+\.|clean\s+-[a-z]*f|stash\b)/ }, + { rule: 'auto-approve-gate', re: /auto-?approv\w*/i }, + { rule: 'hand-edit-status', re: /(hand|manual\w*)[- ]edit\w*\s+status\.yaml|edited?\s+status\.yaml\s+(directly|by hand)/i }, + { rule: 'afx-from-worktree', re: /\bafx\s+spawn\b[^\n]*\bfrom\s+(inside\s+)?(a\s+)?worktree/i }, + { rule: 'kill-shellper', re: /kill\w*\s+[^\n]{0,40}shellper/i }, + { rule: 'restart-tower', re: /\b(restart|stop)\w*\s+tower\b/i }, +]; + +/** Phrases indicating the surrounding text DESCRIBES a rule rather than reports a breach. */ +const DOCUMENTATION_MARKERS = + /\b(never|do not|don'?t|must not|forbidden|prohibited|avoid|rule|policy|reminder|guard|prevent)\b/i; + +// --------------------------------------------------------------------------- +// Helpers +// --------------------------------------------------------------------------- + +function mean(xs: number[]): number { + return xs.length ? xs.reduce((a, b) => a + b, 0) / xs.length : 0; +} +function median(xs: number[]): number { + if (!xs.length) return 0; + const s = [...xs].sort((a, b) => a - b); + const m = Math.floor(s.length / 2); + return s.length % 2 ? s[m] : (s[m - 1] + s[m]) / 2; +} +function round2(n: number): number { + return Math.round(n * 100) / 100; +} + +function walk(dir: string, pred: (p: string) => boolean): string[] { + if (!fs.existsSync(dir)) return []; + const out: string[] = []; + for (const e of fs.readdirSync(dir, { withFileTypes: true })) { + const p = path.join(dir, e.name); + if (e.isDirectory()) out.push(...walk(p, pred)); + else if (pred(p)) out.push(p); + } + return out; +} + +// --------------------------------------------------------------------------- +// Metric collection +// --------------------------------------------------------------------------- + +/** + * B1/B2/B4 — CMAP verdicts and review rounds. + * + * Sample: projects whose `status.yaml` has a non-empty `history`. In practice + * this is SPIR (and one legacy `spider`) — porch only persists per-plan-phase + * review history for protocols that loop that way, so pir/bugfix/air contribute + * nothing. Sorted for deterministic output. + */ +function collectReviewMetrics(root: string, excludeProjects: string[]) { + const projDir = path.join(root, 'codev', 'projects'); + const files = fs.existsSync(projDir) + ? fs + .readdirSync(projDir) + .filter((d) => !excludeProjects.includes(d)) + .map((d) => path.join(projDir, d, 'status.yaml')) + .filter((f) => fs.existsSync(f)) + .sort() + : []; + + const verdictCounts: Record = {}; + const phaseRounds: number[] = []; + const projectRounds: number[] = []; + const sampleProjects: string[] = []; + + for (const f of files) { + let doc: StatusYaml; + try { + doc = yaml.load(fs.readFileSync(f, 'utf-8')) as StatusYaml; + } catch { + continue; // a malformed status file must not abort the measurement + } + const history = doc?.history ?? []; + if (!history.length) continue; + + sampleProjects.push(path.basename(path.dirname(f))); + + const byPhase = new Map(); + for (const item of history) { + for (const r of item.reviews ?? []) { + const v = r.verdict ?? 'UNKNOWN'; + verdictCounts[v] = (verdictCounts[v] ?? 0) + 1; + } + const ph = item.plan_phase ?? '__none__'; + byPhase.set(ph, Math.max(byPhase.get(ph) ?? 0, item.iteration ?? 0)); + } + const rounds = [...byPhase.values()]; + phaseRounds.push(...rounds); + projectRounds.push(rounds.reduce((a, b) => a + b, 0)); + } + + const total = Object.values(verdictCounts).reduce((a, b) => a + b, 0); + return { + verdictCounts, + total, + rcRate: total ? (verdictCounts['REQUEST_CHANGES'] ?? 0) / total : 0, + phaseRounds, + projectRounds, + sampleProjects: sampleProjects.sort(), + }; +} + +/** + * B3 — candidate scar-rule violations mined from committed prose. + * + * Scans `codev/reviews/*.md` and `codev/state/*_thread.md`. Emits EXCERPTS, not + * just counts: the mining cannot distinguish "we violated this" from "never do + * this", so every hit is a candidate requiring human adjudication. Lines that + * look like rule documentation are filtered out to cut the obvious noise, but + * the filter is conservative by design — it is better to hand a human a false + * positive than to silently drop a real breach. + */ +function collectScarHits(root: string, excludeBasenamePrefixes: string[]) { + const excluded = (p: string) => + excludeBasenamePrefixes.some((pre) => path.basename(p).startsWith(pre)); + const targets = [ + ...walk(path.join(root, 'codev', 'reviews'), (p) => p.endsWith('.md')), + ...walk(path.join(root, 'codev', 'state'), (p) => p.endsWith('_thread.md')), + ] + .filter((p) => !excluded(p)) + .sort(); + + const hits: ScarHit[] = []; + for (const file of targets) { + const lines = fs.readFileSync(file, 'utf-8').split('\n'); + lines.forEach((line, i) => { + if (DOCUMENTATION_MARKERS.test(line)) return; // prescriptive, not a breach report + for (const { rule, re } of SCAR_PATTERNS) { + if (re.test(line)) { + hits.push({ + rule, + file: path.relative(root, file), + line: i + 1, + excerpt: line.trim().slice(0, 200), + }); + break; + } + } + }); + } + return { hits, filesScanned: targets.length }; +} + +export interface MeasureOptions { + /** + * Project directory names to exclude from B1/B2/B4. + * + * The baseline is defined as the PRE-PROJECT state (spec 1252, M12): the + * project doing the measuring must not contaminate its own baseline. Its + * review verdicts accumulate in its status.yaml *while it runs*, so an + * unexcluded rerun mid-project would drift from the committed artifact — + * which is exactly how this option came to exist (the Phase-1 reproduction + * test failed the moment this project's own iter-1 review landed). + */ + excludeProjects?: string[]; + /** + * Basename prefixes excluded from B3's prose scan, for the same reason: this + * project's own thread/review discuss scar rules at length and grow with + * every phase, so scanning them makes B3 non-reproducible AND self-inflating. + */ + excludeFilePrefixes?: string[]; +} + +/** This project's own artifacts — excluded from its own baseline by default. */ +export const SELF_PROJECT_DIR = '1252-prompt-architecture-single-own'; +export const SELF_FILE_PREFIXES = ['spir-1252_', '1252-']; + +/** Collect all behavioural metrics for a repo root. Deterministic over B1–B4. */ +export function measureBehavior(root: string, opts: MeasureOptions = {}): BehaviorMetrics { + const rv = collectReviewMetrics(root, opts.excludeProjects ?? [SELF_PROJECT_DIR]); + const scar = collectScarHits(root, opts.excludeFilePrefixes ?? SELF_FILE_PREFIXES); + return { + b1_requestChangesRate: round2(rv.rcRate * 100), + b1_verdictCounts: rv.verdictCounts, + b1_totalVerdicts: rv.total, + b2_roundsPerPhaseMean: round2(mean(rv.phaseRounds)), + b2_roundsPerPhaseMedian: median(rv.phaseRounds), + b2_roundsPerPhaseMax: rv.phaseRounds.length ? Math.max(...rv.phaseRounds) : 0, + b2_phaseCount: rv.phaseRounds.length, + b3_candidateHits: scar.hits, + b3_filesScanned: scar.filesScanned, + b4_roundsPerProjectMean: round2(mean(rv.projectRounds)), + b4_roundsPerProjectMedian: median(rv.projectRounds), + b4_projectCount: rv.projectRounds.length, + sampleProjects: rv.sampleProjects, + }; +} + +// --------------------------------------------------------------------------- +// Rendering +// --------------------------------------------------------------------------- + +/** Render metrics as the committed markdown baseline artifact. */ +export function render(m: BehaviorMetrics): string { + const byRule = new Map(); + for (const h of m.b3_candidateHits) byRule.set(h.rule, (byRule.get(h.rule) ?? 0) + 1); + + const lines: string[] = [ + '# Behavioural metrics (Spec 1252, M12 / Appendix D)', + '', + '## B1 — CMAP verdict distribution', + '', + `Total verdicts: **${m.b1_totalVerdicts}** across ${m.b4_projectCount} projects with review history.`, + '', + '| Verdict | Count | Share |', + '|---|---:|---:|', + ...Object.entries(m.b1_verdictCounts) + .sort((a, b) => b[1] - a[1]) + .map( + ([v, c]) => + `| ${v} | ${c} | ${round2((c / Math.max(m.b1_totalVerdicts, 1)) * 100)}% |` + ), + '', + `**B1 REQUEST_CHANGES rate: ${m.b1_requestChangesRate}%** — the load-bearing metric.`, + '', + '## B2 — review rounds per plan phase', + '', + `mean **${m.b2_roundsPerPhaseMean}**, median ${m.b2_roundsPerPhaseMedian}, max ${m.b2_roundsPerPhaseMax} (n=${m.b2_phaseCount} phases)`, + '', + '> Advisory only. The observed range is too narrow to detect a subtle', + '> regression. Note also that phases advance on builder *rebuttal*, not on', + '> unanimous approval — 0 terminal phases in this corpus end with 3x APPROVE —', + '> so a "rounds to unanimity" metric would never resolve.', + '', + '## B4 — review rounds per project', + '', + `mean **${m.b4_roundsPerProjectMean}**, median ${m.b4_roundsPerProjectMedian} (n=${m.b4_projectCount} projects). Advisory.`, + '', + '## B3 — candidate scar-rule violations', + '', + `Scanned **${m.b3_filesScanned}** files (codev/reviews + codev/state threads).`, + `Candidate hits: **${m.b3_candidateHits.length}**`, + '', + '> **These are CANDIDATES, not findings.** Keyword mining cannot distinguish', + '> "we did this" from "never do this". Every hit requires human adjudication', + '> before the hard rollback trigger fires. B3 is the metric that matters most —', + '> it is the only one that would catch a compressed scar rule losing its force.', + '', + ]; + + if (byRule.size) { + lines.push('| Rule | Candidate hits |', '|---|---:|'); + for (const [r, c] of [...byRule.entries()].sort((a, b) => b[1] - a[1])) { + lines.push(`| ${r} | ${c} |`); + } + lines.push('', '### Excerpts for adjudication', ''); + for (const h of m.b3_candidateHits.slice(0, 50)) { + lines.push(`- \`${h.file}:${h.line}\` **[${h.rule}]** — ${h.excerpt}`); + } + if (m.b3_candidateHits.length > 50) { + lines.push(`- …and ${m.b3_candidateHits.length - 50} more (see --json for the full list)`); + } + lines.push(''); + } + + lines.push( + '## B5 — consult cost/duration', + '', + '**Not captured here.** `consult stats` is a rolling 30-day machine-local DB,', + 'so it is not reproducible from a commit. Capture it separately as advisory', + 'context; it is excluded from T14 determinism and drives no rollback trigger.', + '', + '## Sample provenance', + '', + ...m.sampleProjects.map((p) => `- ${p}`), + '' + ); + return lines.join('\n'); +} + +/** CLI entry, invoked by `packages/codev/scripts/measure-prompt-behavior.ts`. */ +export function runCli(argv: string[], cwd: string): string { + const json = argv.includes('--json'); + const root = argv.find((a) => !a.startsWith('--')) ?? cwd; + const m = measureBehavior(path.resolve(root)); + return json ? JSON.stringify(m, null, 2) : render(m); +} diff --git a/scripts/measure-prompt-surface.sh b/scripts/measure-prompt-surface.sh new file mode 100755 index 000000000..5dda6c769 --- /dev/null +++ b/scripts/measure-prompt-surface.sh @@ -0,0 +1,140 @@ +#!/usr/bin/env bash +# +# measure-prompt-surface.sh — word-count measurement of Codev's prompt surface. +# Spec 1252, criterion M6 / test T1. +# +# Run before any trimming (Phase 1) and again after (Phase 7); the delta is the +# N1 figure. Emits a markdown table on stdout. +# +# The number that matters is ALWAYS-ON: content that enters an agent's context +# whether or not it is needed. Total authored surface is much larger (~150k +# words) but most of it is on-demand and already correctly tiered. +# +# Usage: scripts/measure-prompt-surface.sh [repo-root] +set -euo pipefail + +ROOT="${1:-$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)}" +cd "$ROOT" + +w() { # word count of a file, 0 if absent + [ -f "$1" ] && wc -w < "$1" | tr -d ' ' || echo 0 +} +wdir() { # word count of all .md under a dir, 0 if absent + [ -d "$1" ] && find "$1" -name '*.md' -exec cat {} + 2>/dev/null | wc -w | tr -d ' ' || echo 0 +} +fdir() { # count of .md files under a dir + [ -d "$1" ] && find "$1" -name '*.md' | wc -l | tr -d ' ' || echo 0 +} + +CLAUDE_MD=$(w CLAUDE.md) +AGENTS_MD=$(w AGENTS.md) +ARCH_CRIT=$(w codev/resources/arch-critical.md) +LESS_CRIT=$(w codev/resources/lessons-critical.md) +ARCH=$(w codev/resources/arch.md) +LESS=$(w codev/resources/lessons-learned.md) + +SKEL_W=$(wdir codev-skeleton); SKEL_F=$(fdir codev-skeleton) +SKEL_PROTO_W=$(wdir codev-skeleton/protocols) +SKEL_ROLES_W=$(wdir codev-skeleton/roles) +SHADOW_PROTO_W=$(wdir codev/protocols); SHADOW_PROTO_F=$(fdir codev/protocols) +SHADOW_ROLES_W=$(wdir codev/roles); SHADOW_ROLES_F=$(fdir codev/roles) +SKILLS_W=$(wdir .claude/skills); SKILLS_F=$(fdir .claude/skills) + +# Always-on = what rides into context regardless of need. +# CLAUDE.md already inlines the two hot-tier files, so they are NOT re-added +# here (double-counting them would inflate the baseline and flatter the result). +# AGENTS.md is the byte-identical twin for other tools — one or the other +# loads per session, never both, so it is excluded too. +# +# SPAWN_PROMPT and PHASE_TASK are DERIVED from the artifacts that compose them, +# resolved the way the runtime resolves (tier-2 codev/ first, then the shipped +# skeleton), so Phase-7 trims to any component show up in the rerun. Hardcoding +# the one-off measured values (4891 / 1395) broke reproducibility — a rerun +# after trimming would have reported the pre-trim numbers (caught by Codex at +# the Phase-1 review). +# +# SPAWN_PROMPT proxy = spir builder-prompt.md + spir protocol.md (inlined into +# every spawn prompt). The real spawn prompt adds the issue body (~170 words +# here), which varies per project and is not a trimmable prompt surface, so +# it is deliberately excluded from the proxy. +# +# PHASE_TASK proxy = hot tier (injected into every porch phase prompt) + the +# mean of the porch phase prompts. Task-JSON boilerplate varies per phase and +# is porch code, not prompt surface. +resolve() { # two-tier resolve: codev/ wins, else skeleton + if [ -f "codev/$1" ]; then echo "codev/$1" + elif [ -f "codev-skeleton/$1" ]; then echo "codev-skeleton/$1" + else echo /dev/null; fi +} +PHASE_ITERS="${PHASE_ITERS:-10}" +# SERVED words, not authored: {{> partials/...}} includes are expanded into +# every spawn prompt, so the proxy must count them or dedup-by-include would +# claim phantom always-on savings (it changes authored ownership, not served +# bytes). +expanded_w() { # file + partial includes, two levels deep (partials may nest) + local f="$1"; local total; total=$(w "$f") + for inc in $(grep -o '{{> *[^} ]*' "$f" 2>/dev/null | sed 's/{{> *//'); do + local ip; ip="$(resolve "$inc")" + total=$(( total + $(w "$ip") )) + for inc2 in $(grep -o '{{> *[^} ]*' "$ip" 2>/dev/null | sed 's/{{> *//'); do + total=$(( total + $(w "$(resolve "$inc2")") )) + done + done + echo "$total" +} +SPAWN_BP=$(expanded_w "$(resolve protocols/spir/builder-prompt.md)") +SPAWN_PROTO=$(w "$(resolve protocols/spir/protocol.md)") +SPAWN_PROMPT=$(( SPAWN_BP + SPAWN_PROTO )) +PORCH_PROMPT_MEAN=0 +PORCH_DIR="$( [ -d codev/porch/prompts ] && echo codev/porch/prompts || echo codev-skeleton/porch/prompts )" +if [ -d "$PORCH_DIR" ]; then + PORCH_N=$(find "$PORCH_DIR" -name '*.md' | wc -l | tr -d ' ') + if [ "$PORCH_N" -gt 0 ]; then + PORCH_TOTAL=0 + for pf in "$PORCH_DIR"/*.md; do PORCH_TOTAL=$(( PORCH_TOTAL + $(expanded_w "$pf") )); done + PORCH_PROMPT_MEAN=$(( PORCH_TOTAL / PORCH_N )) + fi +fi +PHASE_TASK=$(( ARCH_CRIT + LESS_CRIT + PORCH_PROMPT_MEAN )) +ALWAYS_ON=$(( CLAUDE_MD + SPAWN_PROMPT + PHASE_TASK * PHASE_ITERS )) + +cat </dev/null || echo n/a)\` + +## Per-surface word counts + +| Surface | Words | Files | Load | +|---|---:|---:|---| +| CLAUDE.md | $CLAUDE_MD | 1 | always-on (hot tier inlined) | +| AGENTS.md | $AGENTS_MD | 1 | always-on twin (other tools) | +| arch-critical.md (HOT) | $ARCH_CRIT | 1 | inlined in CLAUDE.md + every phase prompt | +| lessons-critical.md (HOT) | $LESS_CRIT | 1 | same | +| arch.md (COLD) | $ARCH | 1 | on demand | +| lessons-learned.md (COLD) | $LESS | 1 | on demand | +| codev-skeleton/** | $SKEL_W | $SKEL_F | resolver tier 4 | +| — skeleton protocols/ | $SKEL_PROTO_W | — | | +| — skeleton roles/ | $SKEL_ROLES_W | — | | +| codev/protocols/ (shadow) | $SHADOW_PROTO_W | $SHADOW_PROTO_F | wins over skeleton | +| codev/roles/ (shadow) | $SHADOW_ROLES_W | $SHADOW_ROLES_F | wins over skeleton | +| .claude/skills/** | $SKILLS_W | $SKILLS_F | on demand | + +## Always-on load per builder (derived from resolved artifacts) + + CLAUDE.md $CLAUDE_MD + + spawn prompt (proxy) $SPAWN_PROMPT (builder-prompt.md $SPAWN_BP + protocol.md $SPAWN_PROTO) + + phase task (proxy) x $PHASE_ITERS $(( PHASE_TASK * PHASE_ITERS )) (hot tier $(( ARCH_CRIT + LESS_CRIT )) + porch prompt mean $PORCH_PROMPT_MEAN, per iteration) + ------------------------------ + = ALWAYS_ON $ALWAYS_ON + +Proxies deliberately exclude per-project variable content (issue body, +task-JSON boilerplate) — those are not trimmable prompt surface, and including +them would let a rerun's delta be polluted by whichever issue happened to spawn +the measuring builder. AGENTS.md and the hot-tier files are likewise excluded +from the top-level sum: AGENTS.md is the byte-identical twin (one loads per +session, not both), and the hot-tier files appear inside CLAUDE.md's count +(they ARE counted once per phase task, where porch injects them separately). + +ALWAYS_ON_WORDS=$ALWAYS_ON +EOF