feat: isolate parallel orchestrators by worktree

ARCHITECTURE.md

@@ -196,8 +196,8 @@ kill without a recovery daemon.
 
 | Store | Location | Schema | Written by | Read by |
 |---|---|---|---|---|
-| Write Lease interval | `<git-common-dir>/maestro-write.lock/` | Directory (mutex via `mkdir`); `metadata` file: `token`, `pid`, `process_start`, `job_id`, `session_id`, `started_at`, `started_epoch`, `digest_before`, plus poison fields `quiescence=unconfirmed`, `unconfirmed_job`, `unconfirmed_reason`; `heartbeat` (`token`, `epoch`); staged poison `metadata.new`; generation claim `.reclaim/` | Lease owner; poison stager; reclaimers | Contenders, `--clear-lease`, gate lifecycle |
-| Provenance log | `<git-common-dir>/maestro-provenance.log` | One record per line, appended with `O_NOFOLLOW`, `0600`. `dispatch`/`orphan-adopted` records: `<timestamp> type=dispatch\|orphan-adopted job=<id> session=<id\|unknown> before=<tree-v2:…> after=<tree-v2:…>`; `gap` records: `<timestamp> type=gap prior_job=<id> session=<id\|unknown> expected=<tree-v2:…> observed=<tree-v2:…>` | Lease release (dispatch), acquisition (gap), stale reclaim (orphan-adopted) | Next acquirer (baseline comparison), `provenance_check` |
+| Write Lease interval | `<git-dir>/maestro-write.lock/` | Per-worktree directory (mutex via `mkdir`); `metadata` file: `token`, `pid`, `process_start`, `job_id`, `session_id`, `started_at`, `started_epoch`, `digest_before`, plus poison fields `quiescence=unconfirmed`, `unconfirmed_job`, `unconfirmed_reason`; `heartbeat` (`token`, `epoch`); staged poison `metadata.new`; generation claim `.reclaim/` | Lease owner; poison stager; reclaimers | Contenders in the same worktree, `--clear-lease`, gate lifecycle |
+| Provenance log | `<git-dir>/maestro-provenance.log` | Per-worktree records, one per line, appended with `O_NOFOLLOW`, `0600`. `dispatch`/`orphan-adopted` records: `<timestamp> type=dispatch\|orphan-adopted job=<id> session=<id\|unknown> before=<tree-v3:…> after=<tree-v3:…>`; `gap` records: `<timestamp> type=gap prior_job=<id> session=<id\|unknown> expected=<tree-v3:…> observed=<tree-v3:…>` | Lease release (dispatch), acquisition (gap), stale reclaim (orphan-adopted) | Next acquirer in that worktree (baseline comparison), `provenance_check` |
 | Discussion transcript | `~/.maestro/discussions/<workspace>-<pathhash12>-<slug>.md` | Markdown; turn headers `### Claude (turn N)` / `### Codex (turn N · model=… effort=…)`; sidecar `<T>.state` (`turns`, `awaiting_reply`, `rollback_bytes`); `<T>.lock/` with `metadata` | `discussion-loop.sh` | `discussion-loop.sh`, the orchestrator (relays), users |
 | Direct-edit authorization | `~/.maestro/direct-edit/maestro-direct-<sid>.flag` | Exactly `1\n`, owner-only `0600`, directory `0700` | `orchestrator-inject.mjs` | `orchestrator-gate.mjs` |
 | Session-start preference | `~/.maestro/ask-on-start` | Empty marker file; armed by default on first install, managed by `codex-model-select.sh --ask-on-start on\|off` and removed by uninstall | `install.mjs`, `codex-model-select.sh` | `session-start.mjs`, `uninstall.mjs` |
@@ -260,7 +260,7 @@ alive, or while the generation changed).
 
 ### 3.4 Schema design principles
 
-- **Self-describing records** (`tree-v2:…` digests, `state=…` facts) so a
+- **Self-describing records** (`tree-v3:…` digests, `state=…` facts) so a
   reader can detect "no observation" without guessing.
 - **Atomicity before durability**: every state transition is temp-file +
   rename; poison is staged *before* external cancellation is attempted.
@@ -339,7 +339,7 @@ Knobs are environment variables read at runtime, each validated and
 fail-safed (invalid values warn and fall back, never silently change
 semantics): `MAESTRO_MAX_DISPATCH_SEC` (2400 write / 1200 read), 
 `MAESTRO_VERIFY_TIMEOUT_SEC` (900), `MAESTRO_COMPANION_TIMEOUT_SEC` (120),
-`MAESTRO_DIGEST_TIMEOUT_SEC` (120), `MAESTRO_LOCK_WAIT_SEC` (300, `0` disables),
+`MAESTRO_DIGEST_TIMEOUT_SEC` (120), `MAESTRO_LOCK_WAIT_SEC` (14400, `0` disables),
 `MAESTRO_LOCK_WAIT_POLL_SEC` (5), `MAESTRO_LOCK_HEARTBEAT_INTERVAL_SEC` (20),
 `MAESTRO_LOCK_HEARTBEAT_STALE_SEC` (90), `MAESTRO_SESSION_ID`,
 `MAESTRO_SUITE_TIMEOUT_SEC` (600), `MAESTRO_MAX_ROUNDS` (6),
@@ -482,30 +482,29 @@ documented remedy is to be harder on the diff, not to pretend independence.
 
 ### 7.1 Scaling model
 
-Maestro is **deliberately non-horizontal**: one working tree permits exactly
-one write Lease interval at a time. Concurrency between sessions is handled by
-contention *waiting* (`MAESTRO_LOCK_WAIT_SEC`, default 300s, unordered,
-bounded) and generation-fenced reclaim, never by parallel writers. Vertical
-scaling is by model choice: the orchestrator model and the Codex
-model/effort tiers are runtime-selectable knobs (`codex-model-select.sh`),
-with separate effort tiers for debate vs. implementation.
+Maestro scales horizontally only through isolated Git worktrees: one
+orchestrator task and one write Lease interval per materialized worktree.
+Treehouse supplies and retains those pooled worktrees; Maestro deliberately
+depends only on standard Git `--git-dir` semantics and never parses Treehouse
+state. Sessions inside one worktree remain serialized by an unordered, bounded
+wait (`MAESTRO_LOCK_WAIT_SEC`, default 14400s) and generation-fenced reclaim.
+Separate worktrees may write their own branches concurrently.
 
 What the design optimizes instead:
 
-- **Serial safety** — one writer, bounded waits, no starvation beyond the cap.
+- **Local serial safety** — one writer per worktree, bounded waits, no duplicate dispatch.
+- **Task isolation** — write/job locks and provenance never cross a worktree boundary.
 - **Predictable termination** — every run has a hard ceiling and a
   machine-readable end state.
 - **Cheap re-entry** — a crashed run leaves files, not locks in memory;
   recovery is a documented command.
 
 **Worst-case lease hold.** One Lease interval spans, per iteration: dispatch
 (≤ 2400s) + local verification (≤ 900s) + two bounded tree digests (≤ 120s
-each) — up to `--max-iters` (default 4) iterations. A single run can therefore
-hold the exclusive lease for hours while competitors wait only
-`MAESTRO_LOCK_WAIT_SEC` (default 300s) before hard-`BLOCKED`. That is the
-accepted serialization cost of exclusive ownership: contention is resolved by
-waiting and queueing discipline, never by parallel writers, and the caps make
-the worst case finite.
+each) — up to `--max-iters` (default 4) iterations, or 14,160 seconds. The
+default 14,400-second contention cap covers that bounded run plus 240 seconds
+of orchestration overhead. A poison, malformed owner, or unconfirmed writer
+still fails closed instead of waiting blindly.
 
 ### 7.2 Performance techniques
 
@@ -514,7 +513,7 @@ the worst case finite.
 | Process supervision | Every spawned process runs in its own process group; TERM → 5s grace → KILL; descendants are reaped on every terminal path (verifier, dispatch, tests) |
 | Polling | Poll sleeps clip to the nearest idle or dispatch deadline; status calls reuse a bounded per-call timeout; log-size growth tracked to detect liveness without polling output lines; 4 consecutive empty/malformed statuses fail closed |
 | Budgets | `MAESTRO_MAX_DISPATCH_SEC` hard ceiling (2400s write / 1200s read, explicit values exact); midpoint warning; idle measured as elapsed time since last observed log growth (`--max-idle` default 300s, via bash `SECONDS` — wall-clock-derived, not a true monotonic clock); local verifier deadline 900s in its own group |
-| Tree digest | `git hash-object --no-filters` streams file content through Git itself (no `shasum` dependency); bounded by `MAESTRO_DIGEST_TIMEOUT_SEC` (120s); timeout degrades to `unavailable` (comparison disabled) rather than blocking dispatch; nested worktrees/submodules included, ignored paths excluded on cost grounds |
+| Tree digest | `git hash-object --no-filters` streams current-worktree content through Git itself (no `shasum` dependency); bounded by `MAESTRO_DIGEST_TIMEOUT_SEC` (120s); timeout degrades to `unavailable` (comparison disabled) rather than blocking dispatch; initialized submodules and non-ignored nested repositories included, other linked worktrees excluded |
 | Retry discipline | Loop iterations capped (`--max-iters`, default 4, 0 prohibited); discussion capped at 6 turns with configured retries; never dispatching the identical plan a third time is a rule-level discipline (`rules/orchestrator-implementer.md`), not an enforced technique |
 | I/O | Atomic renames for every state write; xargs-serial hashing preserves order; preview lines diffed against the previous sample to emit only deltas |
 

README.md

@@ -83,13 +83,26 @@ One Implementation run acquires one Lease interval across every Write turn and l
 The loop evaluates `--verify` from the lease repository root, so root-relative
 verification does not depend on the shell directory that launched the loop.
 
-Write contention waits without arrival ordering only while the current lease has a confirmed release path. `MAESTRO_LOCK_WAIT_SEC` caps the wait (default 300 seconds; `0` disables it), and `MAESTRO_LOCK_WAIT_POLL_SEC` controls polling (default 5 seconds, minimum 1); invalid values disable waiting. Progress is emitted on the first wait tick and then at most every 30 seconds. A terminal `held for` value is the owner's lease age; `wait_budget` and `wait_elapsed` identify the contender's actual wait window.
+Write contention waits without arrival ordering only while the current lease has a confirmed release path. `MAESTRO_LOCK_WAIT_SEC` caps the wait (default 14,400 seconds, enough for the bounded worst case of a default Implementation run; `0` disables it), and `MAESTRO_LOCK_WAIT_POLL_SEC` controls polling (default 5 seconds, minimum 1); invalid values disable waiting. Progress is emitted on the first wait tick and then at most every 30 seconds. A terminal `held for` value is the owner's lease age; `wait_budget` and `wait_elapsed` identify the contender's actual wait window.
 
-Foreground write supervisors update a separate lease heartbeat every `MAESTRO_LOCK_HEARTBEAT_INTERVAL_SEC` (default 20 seconds, minimum 1; invalid values use 20). `MAESTRO_LOCK_HEARTBEAT_STALE_SEC` controls when a missed heartbeat is reported (default 90 seconds; `0` disables staleness reporting; invalid values use 90). A stale heartbeat is only a recovery candidate: `--clear-lease` still refuses while the recorded owner process is alive or unidentifiable, or any repository-global companion writer is visible.
+Foreground write supervisors update a separate lease heartbeat every `MAESTRO_LOCK_HEARTBEAT_INTERVAL_SEC` (default 20 seconds, minimum 1; invalid values use 20). `MAESTRO_LOCK_HEARTBEAT_STALE_SEC` controls when a missed heartbeat is reported (default 90 seconds; `0` disables staleness reporting; invalid values use 90). A stale heartbeat is only a recovery candidate: `--clear-lease` still refuses while the recorded owner process is alive or unidentifiable, or any same-worktree companion writer is visible.
 
 `MAESTRO_MAX_DISPATCH_SEC` is a hard ceiling: unset write jobs get 2400 seconds and read-only discussions get 1200; an explicit valid value is used exactly, while invalid input warns and falls back to 1200. Startup consumes this budget, poll sleeps are clipped to the nearest deadline, and one halfway warning continues the same job without claiming progress or creating a checkpoint. Idle time uses elapsed monotonic time rather than configured poll counts. `--max-idle` and `--poll` must be positive integers and are rejected before any lease or task starts. The local verifier has its own process-group deadline (`MAESTRO_VERIFY_TIMEOUT_SEC`, default 900 seconds), and `MAESTRO_COMPANION_TIMEOUT_SEC` bounds each companion call (default 120 seconds). Four consecutive empty **or malformed** statuses, or the hard ceiling during status loss, cancel and fail closed. Read-only status loss consumes its configured retry allowance; idle/deadline cancellation does not. A write cancellation—including one reported externally by the companion—poisons and retains the lease, ends the loop as `BLOCKED`, emits `UNREPORTED_PARTIAL` at the hard ceiling, and never starts a replacement writer. Once no Codex job is writing, recover with `bash hooks/implementer-loop.sh --clear-lease` (installed: `bash ~/.claude/hooks/implementer-loop.sh --clear-lease`). A metadata-less lease younger than five seconds is treated as an owner still initializing, not an orphan to clear.
 
-All Maestro companion dispatches serialize for their full job lifetime on a per-workspace job lock; after confirming a stale lock's recorded job is terminal, recover with `bash hooks/implementer-loop.sh --clear-job-lock` (installed: `bash ~/.claude/hooks/implementer-loop.sh --clear-job-lock`).
+All Maestro companion dispatches serialize for their full job lifetime on a per-worktree job lock; after confirming a stale lock's recorded job is terminal, recover with `bash hooks/implementer-loop.sh --clear-job-lock` (installed: `bash ~/.claude/hooks/implementer-loop.sh --clear-job-lock`).
+
+### Parallel orchestrators with Treehouse
+
+Maestro integrates with [Treehouse](https://github.com/kunchenguid/treehouse),
+created by [Kun Chen](https://github.com/kunchenguid), for pooled Git worktree
+isolation. One orchestrator session owns one task in one materialized worktree.
+For simultaneous terminals:
+
+1. Run `treehouse get` in each terminal; Treehouse opens a different pooled worktree shell.
+2. Start exactly one orchestrator inside that shell and keep all task commands inside it.
+3. Review and land that task before leaving the shell so Treehouse can safely reuse it.
+
+Maestro keys its write lease, companion job lock, and provenance log to that worktree's `--git-dir`, so separate Treehouse worktrees run concurrently. Two orchestrators accidentally started in the same worktree still serialize safely; they are not isolated and must not be treated as parallel. Treehouse owns allocation and return; Maestro never merges, resets, cleans, or returns a worktree. Do not point a second orchestrator at an active Treehouse path or mutate another task's branch.
 
 A separately pinned scout runs cheap read-only repository reconnaissance through that same serialized companion job lock and fails closed when its scout pin is absent or invalid.
 
@@ -202,7 +215,8 @@ Removes only files whose bytes still match the ownership manifest, strips only e
 Stated plainly, because a tool that overstates its guarantees is worse than one that has fewer.
 
 - **The gate is a guardrail, not a boundary.** It is registered for `Edit|Write|MultiEdit` only. `Bash`, MCP tools, and `Workflow`/`Agent` are *not* matched, so a redirect or `sed -i` reaches the tree untouched. Authorization requires a validated session, lives under a private `~/.maestro/direct-edit` directory with owner/mode/content checks, ignores the legacy forgeable `/tmp` marker path, and is revoked by malformed prompt payloads. Scratch/non-code exemptions use canonical existing targets or parents so symlinks cannot change classification; executable files remain gated regardless of extension. The orchestrator not writing source remains a discipline, not a sandbox.
-- **Provenance detection reports, it never attributes.** Each write-lease acquisition hashes actual materialized bytes with Git filters disabled, using Git itself rather than a platform-specific digest utility, across healthy worktrees, initialized submodules, and non-ignored nested repositories. One prunable worktree degrades independently. Baseline records publish before lease handoff, and log publication atomically replaces rather than follows a symlink. A mismatch names an interval—never a writer—and ignored paths remain out of scope on cost grounds. `MAESTRO_DIGEST_TIMEOUT_SEC` bounds each snapshot (default 120); timeout degrades that interval to `unavailable` and disables comparison rather than blocking dispatch. It is not an adversarial control.
+- **Provenance detection reports, it never attributes.** Each write-lease acquisition hashes actual materialized bytes with Git filters disabled, using Git itself rather than a platform-specific digest utility, within the current task worktree, its initialized submodules, and its non-ignored nested repositories. Other linked worktrees stay outside that task's provenance boundary. The scope change is tagged `tree-v3`, so an older repo-wide `tree-v2` record is treated as no observation instead of a false gap. Baseline records publish before lease handoff, and log publication atomically replaces rather than follows a symlink. A mismatch names an interval—never a writer—and ignored paths remain out of scope on cost grounds. `MAESTRO_DIGEST_TIMEOUT_SEC` bounds each snapshot (default 120); timeout degrades that interval to `unavailable` and disables comparison rather than blocking dispatch. It is not an adversarial control.
+- **Worktrees are not OS sandboxes.** Treehouse separates materialized trees, indexes, branches, Maestro locks, and provenance. Linked worktrees still share Git objects and repository-level state; processes, ports, user-level caches, credentials, and external services also remain shared.
 - **Cancellation terminality is upstream.** The companion does not expose the brokered turn's terminal event to Maestro's shell. A cancelled write may therefore leave unreported edits, so Maestro stops and retains the lease instead of guessing that the turn is quiescent.
 - **Same-vendor review.** The orchestrator reviews its own plan's execution.
 - **Model pin depends on config being honored for debate max/ultra.** Model and wrapper-supported efforts are explicit per task. Debate max/ultra rely on the top-level Codex config because the companion cannot express them; implementation therefore rejects max/ultra instead of silently substituting another tier. A fresh unpinned install cannot dispatch until model and effort values are selected.

hooks/lib-job-lock.sh

@@ -11,10 +11,28 @@ MAESTRO_JOB_LOCK_ACQUIRED=0
 export -n MAESTRO_JOB_LOCK_TOKEN MAESTRO_JOB_LOCK_DIR \
   MAESTRO_JOB_LOCK_IDENTITY MAESTRO_JOB_LOCK_ACQUIRED 2>/dev/null || :
 
+maestro_workspace_scope_root() {
+  local dir top parent selected=""
+  dir=$(pwd -P) || return 1
+  while :; do
+    if top=$(git -C "$dir" rev-parse --show-toplevel 2>/dev/null); then
+      top=$(cd "$top" 2>/dev/null && pwd -P) || return 1
+      selected=$top
+      dir=$(dirname "$top")
+    else
+      parent=$(dirname "$dir")
+      [ "$parent" != "$dir" ] || break
+      dir=$parent
+    fi
+  done
+  [ -n "$selected" ] || return 1
+  printf '%s\n' "$selected"
+}
+
 job_lock_path() {
   local workspace git_dir
-  workspace=$(git rev-parse --show-toplevel 2>/dev/null) || return 1
-  git_dir=$(git -C "$workspace" rev-parse --git-common-dir 2>/dev/null) || return 1
+  workspace=$(maestro_workspace_scope_root) || return 1
+  git_dir=$(git -C "$workspace" rev-parse --git-dir 2>/dev/null) || return 1
   case "$git_dir" in
     /*) ;;
     *) git_dir="$workspace/$git_dir" ;;
@@ -486,7 +504,7 @@ job_lock_acquire() { # read|write
   MAESTRO_JOB_LOCK_DIR=$(job_lock_path) || return 3
   metadata="$MAESTRO_JOB_LOCK_DIR/metadata"
 
-  wait_cap=${MAESTRO_LOCK_WAIT_SEC:-300}
+  wait_cap=${MAESTRO_LOCK_WAIT_SEC:-14400}
   wait_poll=${MAESTRO_LOCK_WAIT_POLL_SEC:-5}
   case "$wait_cap" in
     *[!0-9]*)