diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index a980d60..f0c61a8 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -196,8 +196,8 @@ kill without a recovery daemon. | Store | Location | Schema | Written by | Read by | |---|---|---|---|---| -| Write Lease interval | `/maestro-write.lock/` | Directory (mutex via `mkdir`); `metadata` file: `token`, `pid`, `process_start`, `job_id`, `session_id`, `started_at`, `started_epoch`, `digest_before`, plus poison fields `quiescence=unconfirmed`, `unconfirmed_job`, `unconfirmed_reason`; `heartbeat` (`token`, `epoch`); staged poison `metadata.new`; generation claim `.reclaim/` | Lease owner; poison stager; reclaimers | Contenders, `--clear-lease`, gate lifecycle | -| Provenance log | `/maestro-provenance.log` | One record per line, appended with `O_NOFOLLOW`, `0600`. `dispatch`/`orphan-adopted` records: ` type=dispatch\|orphan-adopted job= session= before= after=`; `gap` records: ` type=gap prior_job= session= expected= observed=` | Lease release (dispatch), acquisition (gap), stale reclaim (orphan-adopted) | Next acquirer (baseline comparison), `provenance_check` | +| Write Lease interval | `/maestro-write.lock/` | Per-worktree directory (mutex via `mkdir`); `metadata` file: `token`, `pid`, `process_start`, `job_id`, `session_id`, `started_at`, `started_epoch`, `digest_before`, plus poison fields `quiescence=unconfirmed`, `unconfirmed_job`, `unconfirmed_reason`; `heartbeat` (`token`, `epoch`); staged poison `metadata.new`; generation claim `.reclaim/` | Lease owner; poison stager; reclaimers | Contenders in the same worktree, `--clear-lease`, gate lifecycle | +| Provenance log | `/maestro-provenance.log` | Per-worktree records, one per line, appended with `O_NOFOLLOW`, `0600`. `dispatch`/`orphan-adopted` records: ` type=dispatch\|orphan-adopted job= session= before= after=`; `gap` records: ` type=gap prior_job= session= expected= observed=` | Lease release (dispatch), acquisition (gap), stale reclaim (orphan-adopted) | Next acquirer in that worktree (baseline comparison), `provenance_check` | | Discussion transcript | `~/.maestro/discussions/--.md` | Markdown; turn headers `### Claude (turn N)` / `### Codex (turn N · model=… effort=…)`; sidecar `.state` (`turns`, `awaiting_reply`, `rollback_bytes`); `.lock/` with `metadata` | `discussion-loop.sh` | `discussion-loop.sh`, the orchestrator (relays), users | | Direct-edit authorization | `~/.maestro/direct-edit/maestro-direct-.flag` | Exactly `1\n`, owner-only `0600`, directory `0700` | `orchestrator-inject.mjs` | `orchestrator-gate.mjs` | | Session-start preference | `~/.maestro/ask-on-start` | Empty marker file; armed by default on first install, managed by `codex-model-select.sh --ask-on-start on\|off` and removed by uninstall | `install.mjs`, `codex-model-select.sh` | `session-start.mjs`, `uninstall.mjs` | @@ -260,7 +260,7 @@ alive, or while the generation changed). ### 3.4 Schema design principles -- **Self-describing records** (`tree-v2:…` digests, `state=…` facts) so a +- **Self-describing records** (`tree-v3:…` digests, `state=…` facts) so a reader can detect "no observation" without guessing. - **Atomicity before durability**: every state transition is temp-file + rename; poison is staged *before* external cancellation is attempted. @@ -339,7 +339,7 @@ Knobs are environment variables read at runtime, each validated and fail-safed (invalid values warn and fall back, never silently change semantics): `MAESTRO_MAX_DISPATCH_SEC` (2400 write / 1200 read), `MAESTRO_VERIFY_TIMEOUT_SEC` (900), `MAESTRO_COMPANION_TIMEOUT_SEC` (120), -`MAESTRO_DIGEST_TIMEOUT_SEC` (120), `MAESTRO_LOCK_WAIT_SEC` (300, `0` disables), +`MAESTRO_DIGEST_TIMEOUT_SEC` (120), `MAESTRO_LOCK_WAIT_SEC` (14400, `0` disables), `MAESTRO_LOCK_WAIT_POLL_SEC` (5), `MAESTRO_LOCK_HEARTBEAT_INTERVAL_SEC` (20), `MAESTRO_LOCK_HEARTBEAT_STALE_SEC` (90), `MAESTRO_SESSION_ID`, `MAESTRO_SUITE_TIMEOUT_SEC` (600), `MAESTRO_MAX_ROUNDS` (6), @@ -482,17 +482,18 @@ documented remedy is to be harder on the diff, not to pretend independence. ### 7.1 Scaling model -Maestro is **deliberately non-horizontal**: one working tree permits exactly -one write Lease interval at a time. Concurrency between sessions is handled by -contention *waiting* (`MAESTRO_LOCK_WAIT_SEC`, default 300s, unordered, -bounded) and generation-fenced reclaim, never by parallel writers. Vertical -scaling is by model choice: the orchestrator model and the Codex -model/effort tiers are runtime-selectable knobs (`codex-model-select.sh`), -with separate effort tiers for debate vs. implementation. +Maestro scales horizontally only through isolated Git worktrees: one +orchestrator task and one write Lease interval per materialized worktree. +Treehouse supplies and retains those pooled worktrees; Maestro deliberately +depends only on standard Git `--git-dir` semantics and never parses Treehouse +state. Sessions inside one worktree remain serialized by an unordered, bounded +wait (`MAESTRO_LOCK_WAIT_SEC`, default 14400s) and generation-fenced reclaim. +Separate worktrees may write their own branches concurrently. What the design optimizes instead: -- **Serial safety** — one writer, bounded waits, no starvation beyond the cap. +- **Local serial safety** — one writer per worktree, bounded waits, no duplicate dispatch. +- **Task isolation** — write/job locks and provenance never cross a worktree boundary. - **Predictable termination** — every run has a hard ceiling and a machine-readable end state. - **Cheap re-entry** — a crashed run leaves files, not locks in memory; @@ -500,12 +501,10 @@ What the design optimizes instead: **Worst-case lease hold.** One Lease interval spans, per iteration: dispatch (≤ 2400s) + local verification (≤ 900s) + two bounded tree digests (≤ 120s -each) — up to `--max-iters` (default 4) iterations. A single run can therefore -hold the exclusive lease for hours while competitors wait only -`MAESTRO_LOCK_WAIT_SEC` (default 300s) before hard-`BLOCKED`. That is the -accepted serialization cost of exclusive ownership: contention is resolved by -waiting and queueing discipline, never by parallel writers, and the caps make -the worst case finite. +each) — up to `--max-iters` (default 4) iterations, or 14,160 seconds. The +default 14,400-second contention cap covers that bounded run plus 240 seconds +of orchestration overhead. A poison, malformed owner, or unconfirmed writer +still fails closed instead of waiting blindly. ### 7.2 Performance techniques @@ -514,7 +513,7 @@ the worst case finite. | Process supervision | Every spawned process runs in its own process group; TERM → 5s grace → KILL; descendants are reaped on every terminal path (verifier, dispatch, tests) | | Polling | Poll sleeps clip to the nearest idle or dispatch deadline; status calls reuse a bounded per-call timeout; log-size growth tracked to detect liveness without polling output lines; 4 consecutive empty/malformed statuses fail closed | | Budgets | `MAESTRO_MAX_DISPATCH_SEC` hard ceiling (2400s write / 1200s read, explicit values exact); midpoint warning; idle measured as elapsed time since last observed log growth (`--max-idle` default 300s, via bash `SECONDS` — wall-clock-derived, not a true monotonic clock); local verifier deadline 900s in its own group | -| Tree digest | `git hash-object --no-filters` streams file content through Git itself (no `shasum` dependency); bounded by `MAESTRO_DIGEST_TIMEOUT_SEC` (120s); timeout degrades to `unavailable` (comparison disabled) rather than blocking dispatch; nested worktrees/submodules included, ignored paths excluded on cost grounds | +| Tree digest | `git hash-object --no-filters` streams current-worktree content through Git itself (no `shasum` dependency); bounded by `MAESTRO_DIGEST_TIMEOUT_SEC` (120s); timeout degrades to `unavailable` (comparison disabled) rather than blocking dispatch; initialized submodules and non-ignored nested repositories included, other linked worktrees excluded | | Retry discipline | Loop iterations capped (`--max-iters`, default 4, 0 prohibited); discussion capped at 6 turns with configured retries; never dispatching the identical plan a third time is a rule-level discipline (`rules/orchestrator-implementer.md`), not an enforced technique | | I/O | Atomic renames for every state write; xargs-serial hashing preserves order; preview lines diffed against the previous sample to emit only deltas | diff --git a/README.md b/README.md index 0e3b7f8..de6543d 100644 --- a/README.md +++ b/README.md @@ -83,13 +83,26 @@ One Implementation run acquires one Lease interval across every Write turn and l The loop evaluates `--verify` from the lease repository root, so root-relative verification does not depend on the shell directory that launched the loop. -Write contention waits without arrival ordering only while the current lease has a confirmed release path. `MAESTRO_LOCK_WAIT_SEC` caps the wait (default 300 seconds; `0` disables it), and `MAESTRO_LOCK_WAIT_POLL_SEC` controls polling (default 5 seconds, minimum 1); invalid values disable waiting. Progress is emitted on the first wait tick and then at most every 30 seconds. A terminal `held for` value is the owner's lease age; `wait_budget` and `wait_elapsed` identify the contender's actual wait window. +Write contention waits without arrival ordering only while the current lease has a confirmed release path. `MAESTRO_LOCK_WAIT_SEC` caps the wait (default 14,400 seconds, enough for the bounded worst case of a default Implementation run; `0` disables it), and `MAESTRO_LOCK_WAIT_POLL_SEC` controls polling (default 5 seconds, minimum 1); invalid values disable waiting. Progress is emitted on the first wait tick and then at most every 30 seconds. A terminal `held for` value is the owner's lease age; `wait_budget` and `wait_elapsed` identify the contender's actual wait window. -Foreground write supervisors update a separate lease heartbeat every `MAESTRO_LOCK_HEARTBEAT_INTERVAL_SEC` (default 20 seconds, minimum 1; invalid values use 20). `MAESTRO_LOCK_HEARTBEAT_STALE_SEC` controls when a missed heartbeat is reported (default 90 seconds; `0` disables staleness reporting; invalid values use 90). A stale heartbeat is only a recovery candidate: `--clear-lease` still refuses while the recorded owner process is alive or unidentifiable, or any repository-global companion writer is visible. +Foreground write supervisors update a separate lease heartbeat every `MAESTRO_LOCK_HEARTBEAT_INTERVAL_SEC` (default 20 seconds, minimum 1; invalid values use 20). `MAESTRO_LOCK_HEARTBEAT_STALE_SEC` controls when a missed heartbeat is reported (default 90 seconds; `0` disables staleness reporting; invalid values use 90). A stale heartbeat is only a recovery candidate: `--clear-lease` still refuses while the recorded owner process is alive or unidentifiable, or any same-worktree companion writer is visible. `MAESTRO_MAX_DISPATCH_SEC` is a hard ceiling: unset write jobs get 2400 seconds and read-only discussions get 1200; an explicit valid value is used exactly, while invalid input warns and falls back to 1200. Startup consumes this budget, poll sleeps are clipped to the nearest deadline, and one halfway warning continues the same job without claiming progress or creating a checkpoint. Idle time uses elapsed monotonic time rather than configured poll counts. `--max-idle` and `--poll` must be positive integers and are rejected before any lease or task starts. The local verifier has its own process-group deadline (`MAESTRO_VERIFY_TIMEOUT_SEC`, default 900 seconds), and `MAESTRO_COMPANION_TIMEOUT_SEC` bounds each companion call (default 120 seconds). Four consecutive empty **or malformed** statuses, or the hard ceiling during status loss, cancel and fail closed. Read-only status loss consumes its configured retry allowance; idle/deadline cancellation does not. A write cancellation—including one reported externally by the companion—poisons and retains the lease, ends the loop as `BLOCKED`, emits `UNREPORTED_PARTIAL` at the hard ceiling, and never starts a replacement writer. Once no Codex job is writing, recover with `bash hooks/implementer-loop.sh --clear-lease` (installed: `bash ~/.claude/hooks/implementer-loop.sh --clear-lease`). A metadata-less lease younger than five seconds is treated as an owner still initializing, not an orphan to clear. -All Maestro companion dispatches serialize for their full job lifetime on a per-workspace job lock; after confirming a stale lock's recorded job is terminal, recover with `bash hooks/implementer-loop.sh --clear-job-lock` (installed: `bash ~/.claude/hooks/implementer-loop.sh --clear-job-lock`). +All Maestro companion dispatches serialize for their full job lifetime on a per-worktree job lock; after confirming a stale lock's recorded job is terminal, recover with `bash hooks/implementer-loop.sh --clear-job-lock` (installed: `bash ~/.claude/hooks/implementer-loop.sh --clear-job-lock`). + +### Parallel orchestrators with Treehouse + +Maestro integrates with [Treehouse](https://github.com/kunchenguid/treehouse), +created by [Kun Chen](https://github.com/kunchenguid), for pooled Git worktree +isolation. One orchestrator session owns one task in one materialized worktree. +For simultaneous terminals: + +1. Run `treehouse get` in each terminal; Treehouse opens a different pooled worktree shell. +2. Start exactly one orchestrator inside that shell and keep all task commands inside it. +3. Review and land that task before leaving the shell so Treehouse can safely reuse it. + +Maestro keys its write lease, companion job lock, and provenance log to that worktree's `--git-dir`, so separate Treehouse worktrees run concurrently. Two orchestrators accidentally started in the same worktree still serialize safely; they are not isolated and must not be treated as parallel. Treehouse owns allocation and return; Maestro never merges, resets, cleans, or returns a worktree. Do not point a second orchestrator at an active Treehouse path or mutate another task's branch. A separately pinned scout runs cheap read-only repository reconnaissance through that same serialized companion job lock and fails closed when its scout pin is absent or invalid. @@ -202,7 +215,8 @@ Removes only files whose bytes still match the ownership manifest, strips only e Stated plainly, because a tool that overstates its guarantees is worse than one that has fewer. - **The gate is a guardrail, not a boundary.** It is registered for `Edit|Write|MultiEdit` only. `Bash`, MCP tools, and `Workflow`/`Agent` are *not* matched, so a redirect or `sed -i` reaches the tree untouched. Authorization requires a validated session, lives under a private `~/.maestro/direct-edit` directory with owner/mode/content checks, ignores the legacy forgeable `/tmp` marker path, and is revoked by malformed prompt payloads. Scratch/non-code exemptions use canonical existing targets or parents so symlinks cannot change classification; executable files remain gated regardless of extension. The orchestrator not writing source remains a discipline, not a sandbox. -- **Provenance detection reports, it never attributes.** Each write-lease acquisition hashes actual materialized bytes with Git filters disabled, using Git itself rather than a platform-specific digest utility, across healthy worktrees, initialized submodules, and non-ignored nested repositories. One prunable worktree degrades independently. Baseline records publish before lease handoff, and log publication atomically replaces rather than follows a symlink. A mismatch names an interval—never a writer—and ignored paths remain out of scope on cost grounds. `MAESTRO_DIGEST_TIMEOUT_SEC` bounds each snapshot (default 120); timeout degrades that interval to `unavailable` and disables comparison rather than blocking dispatch. It is not an adversarial control. +- **Provenance detection reports, it never attributes.** Each write-lease acquisition hashes actual materialized bytes with Git filters disabled, using Git itself rather than a platform-specific digest utility, within the current task worktree, its initialized submodules, and its non-ignored nested repositories. Other linked worktrees stay outside that task's provenance boundary. The scope change is tagged `tree-v3`, so an older repo-wide `tree-v2` record is treated as no observation instead of a false gap. Baseline records publish before lease handoff, and log publication atomically replaces rather than follows a symlink. A mismatch names an interval—never a writer—and ignored paths remain out of scope on cost grounds. `MAESTRO_DIGEST_TIMEOUT_SEC` bounds each snapshot (default 120); timeout degrades that interval to `unavailable` and disables comparison rather than blocking dispatch. It is not an adversarial control. +- **Worktrees are not OS sandboxes.** Treehouse separates materialized trees, indexes, branches, Maestro locks, and provenance. Linked worktrees still share Git objects and repository-level state; processes, ports, user-level caches, credentials, and external services also remain shared. - **Cancellation terminality is upstream.** The companion does not expose the brokered turn's terminal event to Maestro's shell. A cancelled write may therefore leave unreported edits, so Maestro stops and retains the lease instead of guessing that the turn is quiescent. - **Same-vendor review.** The orchestrator reviews its own plan's execution. - **Model pin depends on config being honored for debate max/ultra.** Model and wrapper-supported efforts are explicit per task. Debate max/ultra rely on the top-level Codex config because the companion cannot express them; implementation therefore rejects max/ultra instead of silently substituting another tier. A fresh unpinned install cannot dispatch until model and effort values are selected. diff --git a/hooks/lib-job-lock.sh b/hooks/lib-job-lock.sh index 1ca4500..455fb9f 100644 --- a/hooks/lib-job-lock.sh +++ b/hooks/lib-job-lock.sh @@ -11,10 +11,28 @@ MAESTRO_JOB_LOCK_ACQUIRED=0 export -n MAESTRO_JOB_LOCK_TOKEN MAESTRO_JOB_LOCK_DIR \ MAESTRO_JOB_LOCK_IDENTITY MAESTRO_JOB_LOCK_ACQUIRED 2>/dev/null || : +maestro_workspace_scope_root() { + local dir top parent selected="" + dir=$(pwd -P) || return 1 + while :; do + if top=$(git -C "$dir" rev-parse --show-toplevel 2>/dev/null); then + top=$(cd "$top" 2>/dev/null && pwd -P) || return 1 + selected=$top + dir=$(dirname "$top") + else + parent=$(dirname "$dir") + [ "$parent" != "$dir" ] || break + dir=$parent + fi + done + [ -n "$selected" ] || return 1 + printf '%s\n' "$selected" +} + job_lock_path() { local workspace git_dir - workspace=$(git rev-parse --show-toplevel 2>/dev/null) || return 1 - git_dir=$(git -C "$workspace" rev-parse --git-common-dir 2>/dev/null) || return 1 + workspace=$(maestro_workspace_scope_root) || return 1 + git_dir=$(git -C "$workspace" rev-parse --git-dir 2>/dev/null) || return 1 case "$git_dir" in /*) ;; *) git_dir="$workspace/$git_dir" ;; @@ -486,7 +504,7 @@ job_lock_acquire() { # read|write MAESTRO_JOB_LOCK_DIR=$(job_lock_path) || return 3 metadata="$MAESTRO_JOB_LOCK_DIR/metadata" - wait_cap=${MAESTRO_LOCK_WAIT_SEC:-300} + wait_cap=${MAESTRO_LOCK_WAIT_SEC:-14400} wait_poll=${MAESTRO_LOCK_WAIT_POLL_SEC:-5} case "$wait_cap" in *[!0-9]*) diff --git a/hooks/lib-write-lease.sh b/hooks/lib-write-lease.sh index 0f46364..7e96fae 100755 --- a/hooks/lib-write-lease.sh +++ b/hooks/lib-write-lease.sh @@ -26,14 +26,13 @@ _MAESTRO_WRITE_LEASE_RETAIN=0 export -n MAESTRO_LOCK_TOKEN MAESTRO_LOCK_DIR MAESTRO_LOCK_IDENTITY \ MAESTRO_LOCK_ACQUIRED 2>/dev/null || : repo_digest() { - local inside worktrees roots root_list digest material tracked untracked paths entries - local regular_paths hashes link_output link_rc path type mode contents worktree nested_list candidate nested_top + local inside worktree roots root_list digest material tracked untracked paths entries + local regular_paths hashes link_output link_rc path type mode contents nested_list candidate nested_top local root_queue discovered root inside=$(git rev-parse --is-inside-work-tree 2>/dev/null) || return 1 [ "$inside" = "true" ] || return 1 - worktrees=$(git worktree list --porcelain 2>/dev/null) || return 1 - worktrees=$(printf '%s\n' "$worktrees" | sed -n 's/^worktree //p' | LC_ALL=C sort) || return 1 - [ -n "$worktrees" ] || return 1 + worktree=$(write_lock_scope_root) || return 1 + [ -d "$worktree" ] || return 1 root_list=$(mktemp "${TMPDIR:-/tmp}/maestro-repo-roots.XXXXXX") || return 1 root_queue="${root_list}.queue" discovered="${root_list}.discovered" @@ -41,15 +40,7 @@ repo_digest() { if ! ( : > "$root_list" || exit 1 : > "$root_queue" || exit 1 - while IFS= read -r worktree; do - [ -n "$worktree" ] || continue - if [ ! -d "$worktree" ] || - ! git -C "$worktree" rev-parse --is-inside-work-tree >/dev/null 2>&1; then - printf 'MAESTRO_DIGEST: skipping invalid/prunable worktree record: %s\n' "$worktree" >&2 - continue - fi - printf '%s\n' "$worktree" >> "$root_queue" || exit 1 - done <<< "$worktrees" + printf '%s\n' "$worktree" >> "$root_queue" || exit 1 exec 9< "$root_queue" || exit 1 while IFS= read -r root <&9; do @@ -189,7 +180,7 @@ repo_digest() { fi rm -f "$material" "$tracked" "$untracked" "$paths" "$entries" "$regular_paths" "$hashes" [ -n "$digest" ] || return 1 - printf 'tree-v2:%s\n' "$digest" + printf 'tree-v3:%s\n' "$digest" } repo_digest_bounded() { @@ -220,34 +211,19 @@ repo_digest_bounded() { repo_digest_is_observed() { case "$1" in - tree-v2:*) return 0 ;; + tree-v3:*) return 0 ;; *) return 1 ;; esac } write_lock_scope_root() { - local dir top parent selected="" - dir=$(pwd -P) || return 1 - while :; do - if top=$(git -C "$dir" rev-parse --show-toplevel 2>/dev/null); then - top=$(cd "$top" 2>/dev/null && pwd -P) || return 1 - selected=$top - dir=$(dirname "$top") - else - parent=$(dirname "$dir") - [ "$parent" != "$dir" ] || break - dir=$parent - fi - done - [ -n "$selected" ] || return 1 - printf '%s\n' "$selected" + maestro_workspace_scope_root } write_lock_path() { local workspace git_dir if workspace=$(write_lock_scope_root); then - if { git_dir=$(git -C "$workspace" rev-parse --git-common-dir 2>/dev/null) && [ -n "$git_dir" ]; } || - git_dir=$(git -C "$workspace" rev-parse --git-dir 2>/dev/null); then + if git_dir=$(git -C "$workspace" rev-parse --git-dir 2>/dev/null); then case "$git_dir" in /*) ;; *) git_dir="$workspace/$git_dir" ;; @@ -593,9 +569,9 @@ write_lock_acquire() { MAESTRO_LOCK_DIR=$(write_lock_path) || return 3 metadata="$MAESTRO_LOCK_DIR/metadata" - wait_cap=${MAESTRO_LOCK_WAIT_SEC:-300} + wait_cap=${MAESTRO_LOCK_WAIT_SEC:-14400} wait_poll=${MAESTRO_LOCK_WAIT_POLL_SEC:-5} - # A typo must fail fast: a 300-second fallback would turn bad input into a five-minute stall. + # A typo must fail fast: silently substituting a long default would hide bad input. case "$wait_cap" in *[!0-9]*) progress "MAESTRO_LOCK: invalid MAESTRO_LOCK_WAIT_SEC=$wait_cap; waiting disabled" diff --git a/rules/orchestrator-implementer.md b/rules/orchestrator-implementer.md index 0082356..3495249 100644 --- a/rules/orchestrator-implementer.md +++ b/rules/orchestrator-implementer.md @@ -18,6 +18,16 @@ Effort guide: minimal/low for quick mechanical work, medium for default implemen ## Dispatch +### Parallel terminals + +One orchestrator session owns one task and one worktree. For parallel work, +run `treehouse get` in each terminal, then start exactly one orchestrator inside +each leased shell. Never enter another active task's Treehouse path, switch or +rewrite its branch, or dispatch two independent plans from one worktree. +Treehouse owns allocation and return; Maestro never cleans, resets, merges, or +returns the worktree. If two orchestrators share a worktree by mistake, Maestro +serializes their jobs; this is safe fallback, not task isolation. + ``` # Claude Bash tool: run_in_background: true bash ~/.claude/hooks/implementer-loop.sh --plan --verify "" [--max-iters 4] # default @@ -34,9 +44,9 @@ Write access is real — Codex edits your working tree directly. Scope the plan **Orchestrator patience is mandatory.** After dispatching a write-mode job, yield immediately. Perform no workspace reads, verification commands, or diff review until that job reports completion. Reading a tree while Codex is still writing reviews a moving target and silently defeats the loop's final gate. -One Implementation run holds one Lease interval across every Write turn and local Verification transaction. Its ownership domain spans linked worktrees and superproject/submodule overlaps. Contention waits up to `MAESTRO_LOCK_WAIT_SEC` (default 300 seconds, `0` disables waiting) only while the Lease interval has a confirmed release path; each sleep is clipped to the remaining cap. Stale reclaim is token-conditioned, and repository safety queries deliberately ignore companion session filtering and accept compact or pretty valid JSON only. A stale heartbeat is only a recovery candidate: `--clear-lease` still refuses while the recorded owner process is alive or unidentifiable, or any repository-global companion writer is visible. A metadata-less lock younger than five seconds may still be in its atomic-publication window and is not clearable. Never break the lock manually; if cancellation left quiescence unconfirmed, first prove no Codex job is writing, then run `bash ~/.claude/hooks/implementer-loop.sh --clear-lease`. +One Implementation run holds one Lease interval across every Write turn and local Verification transaction. Its ownership domain is one materialized worktree, including superproject/submodule overlaps; linked worktrees have independent domains. Contention waits up to `MAESTRO_LOCK_WAIT_SEC` (default 14,400 seconds, `0` disables waiting) only while the Lease interval has a confirmed release path; each sleep is clipped to the remaining cap. The default covers the bounded worst case of a default Implementation run, so a healthy holder does not require a manual requeue. Stale reclaim is token-conditioned, and repository safety queries deliberately ignore companion session filtering and accept compact or pretty valid JSON only. A stale heartbeat is only a recovery candidate: `--clear-lease` still refuses while the recorded owner process is alive or unidentifiable, or any same-worktree companion writer is visible. A metadata-less lock younger than five seconds may still be in its atomic-publication window and is not clearable. Never break the lock manually; if cancellation left quiescence unconfirmed, first prove no Codex job is writing, then run `bash ~/.claude/hooks/implementer-loop.sh --clear-lease`. -All Maestro companion dispatches—Write turns, debates, and future Read adapters—serialize on a per-workspace job lock for the companion job's full lifetime. Recover a stale lock with `bash ~/.claude/hooks/implementer-loop.sh --clear-job-lock`; direct non-Maestro companion launches bypass this mutex because it is a Maestro convention, not a companion capability. +All Maestro companion dispatches—Write turns, debates, and future Read adapters—serialize on a per-worktree job lock for the companion job's full lifetime. Recover a stale lock with `bash ~/.claude/hooks/implementer-loop.sh --clear-job-lock`; direct non-Maestro companion launches bypass this mutex because it is a Maestro convention, not a companion capability. **Scout** is read-only repository reconnaissance on a separately pinned small model. Pin it with `bash ~/.claude/hooks/codex-model-select.sh --scout ` and dispatch it with `bash ~/.claude/hooks/scout.sh --query `; it serializes on the companion job lock and fails closed when unpinned. @@ -187,11 +197,11 @@ One hole, now version-conditional: - Fixed on Claude Code versions whose subagent PreToolUse payloads carry `agent_id`/`agent_type`: the gate refuses the override when either is present. On older versions that omit the fields, the original hole remains — there is no cross-version schema guarantee. -Fixed: lease scope now selects the outermost enclosing repository and its `--git-common-dir`, so linked worktrees and superproject/submodule entry points serialize one overlapping writable tree. +Fixed: lease scope selects the outermost enclosing repository and its per-worktree `--git-dir`. Superproject/submodule entry points serialize because they overlap one materialized tree; linked worktrees do not because Treehouse tasks must run independently. **Detection, since prevention is unavailable.** A probe measured it: `Edit` and `Write` are blocked, while a `Bash` redirect, `sed -i`, and `git commit` all reached the tree and moved `HEAD`. Prevention would mean enumerating an unbounded set of write paths, so Maestro compares state instead — path-agnostic, and indifferent to whether bytes arrived via `Edit`, `Bash`, an MCP tool, or a workflow agent. -Each write-mode acquisition digests the **materialized tree**—path, entry type, Git-visible executable mode, and actual bytes with clean filters disabled—for tracked and non-ignored untracked files across healthy linked worktrees, initialized submodules, and non-ignored nested repositories. Invalid/prunable worktree records are skipped independently instead of disabling every healthy root. Git itself hashes the final stream, so minimal Linux hosts do not need `shasum`. `MAESTRO_DIGEST_TIMEOUT_SEC` bounds a snapshot (default 120); expiry records `unavailable` and disables that comparison rather than blocking dispatch. Baseline/log publication stays inside the lease-generation claim and atomically replaces, never follows, a provenance-log symlink. `HEAD` and the index are deliberately *not* covered. A mismatch prints one line before the job starts: +Each write-mode acquisition digests the current task's **materialized tree**—path, entry type, Git-visible executable mode, and actual bytes with clean filters disabled—for tracked and non-ignored untracked files in that worktree, its initialized submodules, and its non-ignored nested repositories. Other linked worktrees are deliberately excluded. Git itself hashes the final stream, so minimal Linux hosts do not need `shasum`. `MAESTRO_DIGEST_TIMEOUT_SEC` bounds a snapshot (default 120); expiry records `unavailable` and disables that comparison rather than blocking dispatch. Baseline/log publication stays inside the lease-generation claim and atomically replaces, never follows, a provenance-log symlink. `HEAD` and the index are deliberately *not* covered. A mismatch prints one line before the job starts: ``` PROVENANCE: BASELINE GAP — tree at acquisition differs from the prior completed snapshot (prior_job=…, expected=…, observed=…); author unknown @@ -199,9 +209,9 @@ PROVENANCE: BASELINE GAP — tree at acquisition differs from the prior complete Content, not refs, is what the next job reads — and anchoring to refs made the loop's own prescribed step fire the alarm. Reviewing a dispatch and committing it moved the digest without changing a byte on disk, so every round reported a gap: a 100% false-positive rate on the normal path, which is the alarm fatigue this design rejected when it ruled out sticky warnings. So committing does not move the digest; a `checkout`, `reset`, or stray `sed -i` that changes content still does. A history rewrite that preserves file content is invisible, which is intended. -The digest value is self-describing (`tree-v2:…`). A reader takes the **newest** record and *then* inspects the prefix: an unrecognised or `unavailable` value means **no observation** — never equal, never unequal. Selecting records by prefix instead would skip past newer records to a stale comparable one and manufacture the gap this exists to avoid. One consequence, and it is correct: the first dispatch after a digest-version change finds no comparable baseline, says nothing, and establishes a new one. +The digest value is self-describing (`tree-v3:…`). Version 3 marks the per-worktree scope; prior repo-wide `tree-v2` records are intentionally incompatible. A reader takes the **newest** record and *then* inspects the prefix: an unrecognised or `unavailable` value means **no observation** — never equal, never unequal. Selecting records by prefix instead would skip past newer records to a stale comparable one and manufacture the gap this exists to avoid. One consequence, and it is correct: the first dispatch after a digest-version change finds no comparable baseline, says nothing, and establishes a new one. -Read a gap as *state diverged*, never as *the orchestrator cheated*. Lease metadata delimits an interval; it never identifies which process performed the write. The gap is recorded to `/maestro-provenance.log` and the observed state adopted, so it reports once rather than alarming forever — it does not block, and it never changes `LOOP_STATE` or an exit code. +Read a gap as *state diverged*, never as *the orchestrator cheated*. Lease metadata delimits an interval; it never identifies which process performed the write. The gap is recorded to the current worktree's `/maestro-provenance.log` and the observed state adopted, so it reports once rather than alarming forever — it does not block, and it never changes `LOOP_STATE` or an exit code. A cancelled job may have left edits it never reported, so the tree can contain work with no report describing it. The provenance digest proves that the materialized tree changed; it never proves the job's intent completed. diff --git a/tests/bounded-calls.sh b/tests/bounded-calls.sh index 156aa73..2c5ce54 100755 --- a/tests/bounded-calls.sh +++ b/tests/bounded-calls.sh @@ -294,7 +294,7 @@ t5_repo_digest_survives_refactor() { rc=$? [ "$rc" -eq 0 ] || { echo "rc=$rc want 0"; return 1; } case "$output" in - tree-v2:*) ;; + tree-v3:*) ;; *) echo "digest=$output"; return 1 ;; esac } @@ -592,7 +592,7 @@ check t1_hanging_status_is_bounded "hanging status honors the companion timeout" check t2_timeout_reaps_process_group "timeout returns 125 and reaps the process group" check t3_run_bounded_returns_wrapped_rc "bounded runner preserves stdout and command rc" check t4_invalid_companion_timeout_falls_back "invalid companion timeout falls back to 120s" -check t5_repo_digest_survives_refactor "bounded repository digest still returns tree-v2" +check t5_repo_digest_survives_refactor "bounded repository digest still returns tree-v3" check t6_poll_hanging_status_is_bounded "poll loop bounds repeated hanging statuses" check t7_fast_status_has_no_one_second_floor "five fast status calls finish under two seconds" check t8_explicit_read_ignores_inherited_lease "explicit read mode ignores inherited Lease interval state" diff --git a/tests/commit-invariance.sh b/tests/commit-invariance.sh index 61c6dbe..bcb7694 100755 --- a/tests/commit-invariance.sh +++ b/tests/commit-invariance.sh @@ -44,17 +44,17 @@ O2=$(run "$D/x" 'unset MAESTRO_LOCK_TOKEN; write_lock_acquire job-two') [ "$(gaps "$O2")" -eq 1 ] || { echo "FAIL(2): unattributed redirect not reported (gaps=$(gaps "$O2"))"; exit 1; } echo " ok" -echo "== 3. migration: an unrecognised baseline is no observation, never unequal ==" +echo "== 3. migration: a repo-wide v2 baseline is no observation, never unequal ==" mk "$D/m" LOG="$D/m/.git/maestro-provenance.log" printf '2026-01-01T00:00:00Z type=dispatch job=old-job before=%s after=%s\n' \ - "$(printf 0%.0s $(seq 40))" "$(printf b%.0s $(seq 40))" > "$LOG" + 'tree-v2:old-scope' 'tree-v2:old-scope' > "$LOG" O3=$(run "$D/m" 'unset MAESTRO_LOCK_TOKEN; write_lock_acquire job-one >/dev/null; write_lock_release') -[ "$(gaps "$O3")" -eq 0 ] || { echo "FAIL(3a): old bare record compared as unequal instead of no-observation"; exit 1; } -grep -q 'after=tree-v2:' "$LOG" || { echo "FAIL(3b): no self-describing v2 baseline written" +[ "$(gaps "$O3")" -eq 0 ] || { echo "FAIL(3a): old v2 record compared as unequal instead of no-observation"; exit 1; } +grep -q 'after=tree-v3:' "$LOG" || { echo "FAIL(3b): no self-describing v3 baseline written" echo " log: $(tail -1 "$LOG")"; exit 1; } O4=$(run "$D/m" 'unset MAESTRO_LOCK_TOKEN; write_lock_acquire job-two >/dev/null; write_lock_release') -[ "$(gaps "$O4")" -eq 0 ] || { echo "FAIL(3c): gap on an unchanged tree once a v2 baseline exists"; exit 1; } +[ "$(gaps "$O4")" -eq 0 ] || { echo "FAIL(3c): gap on an unchanged tree once a v3 baseline exists"; exit 1; } printf 'a\nb\n' > "$D/m/s.sh" O5=$(run "$D/m" 'unset MAESTRO_LOCK_TOKEN; write_lock_acquire job-three') [ "$(gaps "$O5")" -eq 1 ] || { echo "FAIL(3d): expected exactly one gap after a one-byte mutation, got $(gaps "$O5")"; exit 1; } @@ -64,7 +64,7 @@ echo "== 4. the digest tracks materialized content, not refs ==" mk "$D/d" d() { ( cd "$D/d" && bash -c "exec 3>&-; set -uo pipefail; . '$LIB'; repo_digest" ) 2>/dev/null; } B=$(d); [ "$(d)" = "$B" ] || { echo "FAIL(4a): digest unstable on an unchanged tree"; exit 1; } -case "$B" in tree-v2:*) ;; *) echo "FAIL(4b): digest is not self-describing: $B"; exit 1 ;; esac +case "$B" in tree-v3:*) ;; *) echo "FAIL(4b): digest is not self-describing: $B"; exit 1 ;; esac ( cd "$D/d" && git add -A && git commit -q --allow-empty -m empty ) [ "$(d)" = "$B" ] || { echo "FAIL(4c): a commit changed the digest"; exit 1; } chmod +x "$D/d/s.sh" diff --git a/tests/job-lock.sh b/tests/job-lock.sh index f292ec3..0183bf2 100644 --- a/tests/job-lock.sh +++ b/tests/job-lock.sh @@ -916,6 +916,33 @@ t22_post_move_unlock_failure_does_not_reclassify_release() ( { echo "post-move unlock failure diagnostic missing"; return 1; } ) +t23_default_wait_covers_a_default_implementation_run() ( + local repo deadline_file before deadline budget output rc + repo="$TEST_ROOT/t23" + new_repo "$repo" + repo=$(cd "$repo" && pwd -P) + deadline_file="$repo/deadline" + cd "$repo" || exit 1 + . "$COMPANION_LIB" + progress_init + job_lock_acquire write >/dev/null 2>&1 || return 1 + unset MAESTRO_JOB_LOCK_TOKEN MAESTRO_LOCK_WAIT_SEC + export MAESTRO_LOCK_WAIT_POLL_SEC=1 + job_lock_wait_tick() { + printf '%s\n' "$1" > "$deadline_file" + return 1 + } + before=$(date +%s) + output=$(job_lock_acquire write 3>&1 >/dev/null 2>&1); rc=$? + [ "$rc" -eq 11 ] || { echo "rc=$rc want 11"; return 1; } + deadline=$(cat "$deadline_file") || return 1 + budget=$((deadline - before)) + [ "$budget" -ge 14399 ] && [ "$budget" -le 14401 ] || + { echo "default wait budget=${budget}s want 14400s"; return 1; } + printf '%s\n' "$output" | grep -q 'companion dispatch blocked' || + { echo "terminal contention diagnostic missing: $output"; return 1; } +) + @@ -946,6 +973,7 @@ check t19_failed_publication_retirement_blocks "failed publication blocks when i check t20_identity_failure_retirement_blocks "identity failure blocks when its generation cannot retire" check t21_token_failure_precedes_creation "token generation fails before canonical creation" check t22_post_move_unlock_failure_does_not_reclassify_release "post-move gate cleanup failure does not reclassify release" +check t23_default_wait_covers_a_default_implementation_run "default job-lock wait covers a default Implementation run" check t8_generation_claim_serializes_reclaimers "generation claims serialize reclaimers and preserve successors" printf '\n=== %d passed, %d failed ===\n' "$PASS" "$FAIL" [ "$FAIL" -eq 0 ] diff --git a/tests/lease.sh b/tests/lease.sh index 3d43143..b6ee81e 100755 --- a/tests/lease.sh +++ b/tests/lease.sh @@ -421,7 +421,7 @@ t17() ( local dir log; dir=$(ws legacy_baseline) git -C "$dir" init -q log="$dir/.git/maestro-provenance.log" - printf '2026-01-01T00:00:00Z type=dispatch job=legacy-job before=tree-v2:old after=tree-v2:old\n' > "$log" + printf '2026-01-01T00:00:00Z type=dispatch job=legacy-job before=tree-v3:old after=tree-v3:old\n' > "$log" cd "$dir" || exit 1; . "$LIB"; progress_init write_lock_acquire task-new00000-cccccc >/dev/null 2>&1 grep -q ' type=gap prior_job=legacy-job ' "$log" || @@ -649,7 +649,7 @@ t27() ( [ "$recorded_token" = "$MAESTRO_LOCK_TOKEN" ] && [ "$recorded_pid" = "$$" ] || return 1 sed -n '1,7p' "$metadata" > "$identity_before" || return 1 - printf 'tree-v2:identity-published\n' + printf 'tree-v3:identity-published\n' } write_lock_acquire task-published-first-aaaaaa >/dev/null 2>&1; rc=$? [ "$rc" -eq 0 ] || { echo "rc=$rc want 0"; return 1; } @@ -666,7 +666,7 @@ t27() ( ''|*[!0-9]*) echo "published pid is not numeric: $owner_pid"; return 1 ;; esac [ "$owner_pid" = "$$" ] || { echo "pid=$owner_pid want $$"; return 1; } - grep -qx 'digest_before=tree-v2:identity-published' "$metadata" || + grep -qx 'digest_before=tree-v3:identity-published' "$metadata" || { echo "digest ran before identity publication"; return 1; } write_lock_release >/dev/null 2>&1 [ ! -d "$dir/.maestro-write.lock" ] || @@ -1980,6 +1980,32 @@ t68_token_failure_precedes_creation() ( { echo "token generation failure created a canonical lock"; return 1; } ) +t69_default_wait_covers_a_default_implementation_run() ( + local dir deadline_file before deadline budget out rc + dir=$(ws default_implementation_wait) + deadline_file="$dir/deadline" + cd "$dir" || exit 1 + . "$LIB" + progress_init + PATH=$(confirmed_ps_path "$dir"); export PATH + write_lock_acquire task-default-wait-holder-aaaaaa >/dev/null 2>&1 || return 1 + unset MAESTRO_LOCK_TOKEN MAESTRO_LOCK_WAIT_SEC + export MAESTRO_LOCK_WAIT_POLL_SEC=1 + write_lock_wait_tick() { + printf '%s\n' "$1" > "$deadline_file" + return 1 + } + before=$(date +%s) + out=$(write_lock_acquire 3>&1 >/dev/null 2>&1); rc=$? + [ "$rc" -eq 11 ] || { echo "rc=$rc want 11"; return 1; } + deadline=$(cat "$deadline_file") || return 1 + budget=$((deadline - before)) + [ "$budget" -ge 14399 ] && [ "$budget" -le 14401 ] || + { echo "default wait budget=${budget}s want 14400s"; return 1; } + printf '%s\n' "$out" | grep -q 'wait_budget=14400s' || + { echo "default wait diagnostic missing 14400s budget: $out"; return 1; } +) + printf '=== Plan F green-phase verification ===\n' @@ -2005,7 +2031,8 @@ for t in t1 t2 t3 t4 t5 t5b t6 t7 t7b t8 t9 t9b t10a t10b t11 t12 t13 t14 t15 t1 t65_poison_finalize_rejects_a_preexisting_same_token_successor \ t66_failed_publication_retirement_blocks \ t67_identity_failure_retirement_blocks \ - t68_token_failure_precedes_creation; do + t68_token_failure_precedes_creation \ + t69_default_wait_covers_a_default_implementation_run; do msg=$($t 2>&1) && ok "$t" || bad "$t" "${msg:-no detail}" done printf '\n=== %d passed, %d failed ===\n' "$PASS" "$FAIL" diff --git a/tests/liveness.sh b/tests/liveness.sh index afdde1b..18d26db 100644 --- a/tests/liveness.sh +++ b/tests/liveness.sh @@ -731,13 +731,13 @@ t8_verifier_boundaries() { { echo "verifier root cwd rc=$WAIT_RC want 0: $(tr '\n' ' ' < "$state/output")"; return 1; } grep -q '^MAESTRO_FINAL: LOOP VERIFIED_DONE rc=0$' "$state/output" || { echo "verifier root cwd final missing: $(tr '\n' ' ' < "$state/output")"; return 1; } - local heartbeat first second + local heartbeat first second verifier_pid verifier_alive repo=$(new_repo verifier-heartbeat-repo) state="$TEST_ROOT/verifier-heartbeat-state" mkdir -p "$state" : > "$state/calls.log" status_empty > "$state/status.json" - verify='sleep 5' + verify="printf '%s\n' \"\${BASHPID:-\$\$}\" > '$state/verifier.pid'; while [ ! -e '$state/allow-verifier-exit' ]; do sleep 0.1; done" set -m ( cd "$repo" && @@ -748,7 +748,7 @@ t8_verifier_boundaries() { MAESTRO_TEST_STATUS="$state/status.json" \ MAESTRO_LOCK_HEARTBEAT_INTERVAL_SEC=1 \ MAESTRO_LOCK_HEARTBEAT_STALE_SEC=2 \ - MAESTRO_VERIFY_TIMEOUT_SEC=8 \ + MAESTRO_VERIFY_TIMEOUT_SEC=30 \ bash "$LOOP" --plan "$TEST_ROOT/plan.md" --verify "$verify" \ --max-iters 1 --poll 1 ) > "$state/output" 2>&1 & @@ -765,15 +765,36 @@ t8_verifier_boundaries() { grep -q 'LOOP: RESULT: DONE on iteration 1' "$state/output" || { echo "verifier did not reach local verification"; return 1; } [ -f "$heartbeat" ] || { echo "verification heartbeat missing"; return 1; } + for _ in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15; do + [ -s "$state/verifier.pid" ] && break + sleep 1 + done + verifier_pid=$(sed -n '1p' "$state/verifier.pid" 2>/dev/null) + case "$verifier_pid" in + ''|*[!0-9]*) : > "$state/allow-verifier-exit"; echo "invalid verifier pid: $verifier_pid"; return 1 ;; + esac first=$(sed -n 's/^epoch=//p' "$heartbeat" | head -1) - sleep 2 - second=$(sed -n 's/^epoch=//p' "$heartbeat" | head -1) + case "$first" in + ''|*[!0-9]*) : > "$state/allow-verifier-exit"; echo "invalid first heartbeat epoch: $first"; return 1 ;; + esac + second=$first + for _ in 1 2 3 4 5 6; do + sleep 1 + second=$(sed -n 's/^epoch=//p' "$heartbeat" | head -1) + case "$second" in + ''|*[!0-9]*) ;; + *) [ "$second" -gt "$first" ] && break ;; + esac + done + kill -0 "$verifier_pid" 2>/dev/null + verifier_alive=$? + : > "$state/allow-verifier-exit" case "$first:$second" in :*|*:|*[!0-9:]*) echo "invalid heartbeat epochs: first=$first second=$second"; return 1 ;; esac [ "$second" -gt "$first" ] || { echo "verification heartbeat did not advance: first=$first second=$second"; return 1; } - kill -0 "$pid" 2>/dev/null || + [ "$verifier_alive" -eq 0 ] || { echo "verifier ended before heartbeat observation"; return 1; } wait_bounded "$pid" 30 [ "$WAIT_TIMED_OUT" -eq 0 ] || { echo "heartbeat verifier exceeded 30s"; return 1; } diff --git a/tests/provenance-edge.sh b/tests/provenance-edge.sh index a797949..42dae95 100644 --- a/tests/provenance-edge.sh +++ b/tests/provenance-edge.sh @@ -47,17 +47,20 @@ t1_clean_filter_cannot_hide_materialized_changes() { [ "$before" != "$after" ] || { echo "clean filter hid a working-tree byte change"; return 1; } } -t2_prunable_worktree_does_not_disable_healthy_root() { - local repo worktree output rc - repo=$(new_repo prunable-main) - worktree="$TEST_ROOT/prunable-wt" - git -C "$repo" worktree add -q "$worktree" -b prunable-branch || return 1 - rm -rf "$worktree" - git -C "$repo" worktree list --porcelain | grep -q '^prunable' || - { echo "git did not mark removed worktree prunable"; return 1; } - output=$(digest "$repo"); rc=$? - [ "$rc" -eq 0 ] || { echo "digest rc=$rc with one healthy root"; return 1; } - case "$output" in tree-v2:*) ;; *) echo "digest=$output"; return 1 ;; esac +t2_linked_worktree_changes_are_isolated() { + local repo worktree main_before main_after worktree_before worktree_after + repo=$(new_repo isolated-main) + worktree="$TEST_ROOT/isolated-wt" + git -C "$repo" worktree add -q "$worktree" -b isolated-branch >/dev/null 2>&1 || return 1 + main_before=$(digest "$repo") || { echo "main digest failed"; return 1; } + worktree_before=$(digest "$worktree") || { echo "worktree digest failed"; return 1; } + printf 'worktree-only change\n' > "$worktree/seed.txt" + main_after=$(digest "$repo") || { echo "main digest after worktree edit failed"; return 1; } + worktree_after=$(digest "$worktree") || { echo "worktree digest after edit failed"; return 1; } + [ "$main_before" = "$main_after" ] || + { echo "another worktree changed the main worktree digest"; return 1; } + [ "$worktree_before" != "$worktree_after" ] || + { echo "worktree-local change was invisible"; return 1; } } t3_untracked_nested_repository_is_observed() { @@ -170,7 +173,7 @@ EOF chmod +x "$shim/shasum" output=$(cd "$repo" && PATH="$shim:$PATH" bash -c "exec 3>&-; set -uo pipefail; . '$LIB'; repo_digest") 2>/dev/null || { echo "digest failed when shasum was unavailable"; return 1; } - case "$output" in tree-v2:*) ;; *) echo "digest=$output"; return 1 ;; esac + case "$output" in tree-v3:*) ;; *) echo "digest=$output"; return 1 ;; esac } t6_provenance_append_never_follows_symlink() { @@ -250,7 +253,7 @@ check() { printf '=== Provenance edge verification ===\n' check t1_clean_filter_cannot_hide_materialized_changes "digest hashes materialized bytes without clean filters" -check t2_prunable_worktree_does_not_disable_healthy_root "one prunable worktree does not disable healthy roots" +check t2_linked_worktree_changes_are_isolated "linked worktree changes remain outside this task's provenance" check t3_untracked_nested_repository_is_observed "untracked nested repositories are observed as materialized roots" check t4_release_publishes_baseline_before_handoff "baseline publication is serialized with lease handoff" check t5_digest_does_not_depend_on_shasum "digest remains available without the non-portable shasum utility" diff --git a/tests/shared-git-dir.sh b/tests/shared-git-dir.sh index 5cf34d9..a4d4e2a 100755 --- a/tests/shared-git-dir.sh +++ b/tests/shared-git-dir.sh @@ -2,6 +2,7 @@ set -uo pipefail ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" LIB="$ROOT/hooks/lib-write-lease.sh" +JOB_LIB="$ROOT/hooks/lib-job-lock.sh" bash -n "$LIB" || { echo "VERIFY FAIL: syntax"; exit 1; } D=$(mktemp -d); trap 'rm -rf "$D"' EXIT @@ -16,9 +17,10 @@ git init -q "$D/repo" && ( A=$(cd "$D/repo" && bash -c "set -uo pipefail; . '$LIB'; write_lock_path") B=$(cd "$D/wt" && bash -c "set -uo pipefail; . '$LIB'; write_lock_path") echo "main: $A"; echo "wt: $B" -[ "$A" = "$B" ] || { echo "VERIFY FAIL: divergent lease paths across worktrees"; exit 1; } +[ "$A" != "$B" ] || { echo "VERIFY FAIL: linked worktrees share a write lease path"; exit 1; } -# cross-worktree contention: holder alive in main checkout, acquirer in worktree must get 11 +# Linked worktrees have separate materialized trees, indexes, and branches. Their +# write leases must therefore be independent so separate terminals can run in parallel. # Write the rc to a file and discard every other stream: progress_init re-points # FD 3 at stdout inside this shell, so capturing stdout would swallow the operator # line into $RC along with the exit code. @@ -26,7 +28,18 @@ echo "main: $A"; echo "wt: $B" cd '$D/wt'; unset MAESTRO_LOCK_TOKEN; write_lock_acquire; echo \$? > '$D/rc.txt'") >/dev/null 2>&1 RC=$(cat "$D/rc.txt" 2>/dev/null) echo "cross-worktree acquire rc=$RC" -[ "$RC" = "11" ] || { echo "VERIFY FAIL: cross-worktree contention rc=$RC want 11"; exit 1; } +[ "$RC" = "0" ] || { echo "VERIFY FAIL: isolated worktree acquire rc=$RC want 0"; exit 1; } + +# Companion job supervision is isolated at the same boundary. A read/debate or +# Write turn in one task worktree must not block a different task worktree. +JA=$(cd "$D/repo" && bash -c "set -uo pipefail; . '$JOB_LIB'; job_lock_path") +JB=$(cd "$D/wt" && bash -c "set -uo pipefail; . '$JOB_LIB'; job_lock_path") +echo "main job lock: $JA"; echo "wt job lock: $JB" +[ "$JA" != "$JB" ] || { echo "VERIFY FAIL: linked worktrees share a companion job lock"; exit 1; } +(cd "$D/repo" && bash -c "set -uo pipefail; . '$JOB_LIB'; progress_init() { :; }; job_lock_acquire write + cd '$D/wt'; unset MAESTRO_JOB_LOCK_TOKEN; job_lock_acquire write; echo \$? > '$D/job-rc.txt'") >/dev/null 2>&1 +JOB_RC=$(cat "$D/job-rc.txt" 2>/dev/null) +[ "$JOB_RC" = "0" ] || { echo "VERIFY FAIL: isolated worktree job-lock acquire rc=$JOB_RC want 0"; exit 1; } # superproject/submodule overlap: the outer repository can edit submodule bytes, # so both entry points must share one ownership domain. @@ -59,8 +72,21 @@ echo "super: $SUPER"; echo "sub: $SUB" SUB_RC=$(cat "$D/sub-rc.txt" 2>/dev/null) [ "$SUB_RC" = 11 ] || { echo "VERIFY FAIL: submodule overlap contention rc=$SUB_RC want 11"; exit 1; } +SUPER_JOB=$(cd "$D/super" && bash -c 'set -uo pipefail; . "$1"; job_lock_path' _ "$JOB_LIB") +SUB_JOB=$(cd "$D/super/dep" && bash -c 'set -uo pipefail; . "$1"; job_lock_path' _ "$JOB_LIB") +echo "super job lock: $SUPER_JOB"; echo "sub job lock: $SUB_JOB" +[ "$SUPER_JOB" = "$SUB_JOB" ] || + { echo "VERIFY FAIL: superproject/submodule job-lock paths differ"; exit 1; } +(cd "$D/super" && bash -c 'set -uo pipefail; . "$1"; progress_init() { :; }; job_lock_acquire write + cd "$2" || exit 1; unset MAESTRO_JOB_LOCK_TOKEN; export MAESTRO_LOCK_WAIT_SEC=0 + job_lock_acquire write; echo $? > "$3"' _ \ + "$JOB_LIB" "$D/super/dep" "$D/sub-job-rc.txt") >/dev/null 2>&1 +SUB_JOB_RC=$(cat "$D/sub-job-rc.txt" 2>/dev/null) +[ "$SUB_JOB_RC" = 11 ] || + { echo "VERIFY FAIL: submodule job-lock contention rc=$SUB_JOB_RC want 11"; exit 1; } + # non-git fallback unchanged N=$(mktemp -d); F=$(cd "$N" && bash -c "set -uo pipefail; . '$LIB'; write_lock_path"); rm -rf "$N" case "$F" in */.maestro-write.lock) ;; *) echo "VERIFY FAIL: non-git fallback = $F"; exit 1;; esac -echo "VERIFY PASS: shared worktree/submodule lease path, contention=11, fallback intact" +echo "VERIFY PASS: worktree-isolated write/job locks, shared submodule scope, fallback intact"